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Abstract 

One  of  the  major  tenets  of  acquisition  reform  is  that  government  agencies 
should  be  as  open  and  forthcoming  as  possible  in  dealing  with  potential  contractors. 
Increased  communication  is  seen  as  one  of  the  keys  to  improve  the  acquisition  process. 
An  important  vehicle  for  government-contractor  communication  is  the  post-award 
debriefing.  A  post-award  debriefing  is  a  meeting  with  government  procurement 
personnel  at  the  request  of  an  unsuccessful  offeror.  In  the  meeting,  the  government 
representatives  explain  why  the  imsuccessful  offeror  did  not  receive  the  contract. 

A  widely  held  belief  in  government  acquisition  circles  is  that  many  protests  have 
been  filed  simply  because  unsuccessful  offerors  have  not  been  fully  informed  of  the  facts 
surrounding  the  contract  award  decision.  Acquisition  reform  legislation  has  attempted  to 
address  this  problem  by  mandating  more  timely  and  complete  post-award  debriefings  that 
provide  substantive  information  concerning  the  basis  for  the  award.  The  theory  is  that 
more  information  in  the  post-award  debriefing  will  lead  to  fewer  protests  by  unsuccessful 
offerors. 

The  primary  objective  of  this  research  is  to  determine  whether  changes  to  the 
post-award  debriefing  process  mandated  by  the  Federal  Acquisition  Streamlining  Act  of 
1994  (FAS A)  have  indeed  resulted  in  fewer  bid  protests.  In  addition,  other  potential 
effects  of  acquisition  reform  legislation  on  the  post-award  debriefing  process  will  be 


investigated. 


AN  EXAMINATION  OF  THE  EFFECTS  OF  THE  FEDERAL  ACQUISITION 
STREAMLINING  ACT  OF  1994  ON  THE  POST-AWARD  DEBRIEFING  PROCESS 

AND  BID  PROTEST  FREQUENCY 

I.  Introduction 

Background 

One  of  the  major  tenets  of  acquisition  reform  is  that  government  agencies  should 
be  as  open  and  forthcoming  as  possible  in  dealing  with  potential  contractors  (Nash  and 
Cibinic,  1994:  33).  An  important  vehicle  for  government-contractor  communication  is 
the  post-award  debriefing.  A  post-award  debriefing  is  a  meeting  with  government 
procurement  personnel  at  the  request  of  an  unsuccessful  offeror.  In  the  meeting,  the 
government  representatives  explain  why  the  unsuccessful  offeror  did  not  receive  the 
contract. 

While  it  has  been  recognized  that  unsuccessful  offerors  (or  “disappointed 
bidders”)  should  have  reasonable  access  to  the  reasons  for  agency  award  decisions, 
agencies  have  not  always  been  forthcoming  in  providing  useful  post-award  debriefing 
information  (Adelson,  1996:  6).  This  situation  came  about  in  large  measure  because  of 
the  rather  restrictive  debriefing  guidance  in  the  Federal  Acquisition  Regulation  (FAR). 
Consequently,  unsuccessful  offerors  sought  relief  through  the  procurement  protest 
process. 

Procurement  protests  play  a  vital  role  in  assuring  fairness  and,  to  the  extent 
possible,  maximum  competition  in  the  federal  acquisition  process  (Bid  Protests  at  GAO: 
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A  Descriptive  Guide,  1996:  1).  They  provide  a  type  of  “policing  function”  to  ensure  that 

government  contracting  agencies  stay  within  prescribed  laws  and  regulations  when 

awarding  contracts  for  goods  and  services.  Nash  and  Cibinic  point  out  that  over  the 

years,  however,  protests  have  evolved  from  a  relatively  simple  investigative  mechanism 

into  full-fledged  litigation,  complete  with  all  the  trappings  of  procedural  due  process 

(Nash  and  Cibinic,  1994:  33).  As  the  process  has  grown  more  complex,  it  also  has  grown 

more  costly.  These  costs  include  not  only  the  expense  of  litigating  protests  in  the  various 

forums  but  also  the  costs  of  agencies’  attempts  to  avoid  protests  through  elaborately 

devised  procurement  techniques  (Nash  and  Cibinic,  1994:  33). 

A  widely  held  belief  in  govenunent  acquisition  circles  is  that  many  protests  have 

been  filed  simply  because  unsuccessful  offerors  have  not  been  fully  informed  of  the  facts 

surrounding  source  selection  decisions — ^the  reasoning  behind  the  decision  to  award  a 

contract  to  a  particular  offeror.  This  belief  was  supported  by  a  survey  by  the  American 

Bar  Association  (ABA)  in  which  respondents  indicated  that  their  lack  of  knowledge  of 

the  significant  facts  of  a  an  award  often  leads  to  a  decision  to  protest  (The  Protest 

Experience,  1989:  79).  Acquisition  reform  legislation  has  attempted  to  address  this 

problem  by  mandating  more  timely  and  complete  post-award  debriefings  that  provide 

substantive  information  concerning  the  basis  for  the  award.  As  Allen  and  others  quote 

from  the  Report  of  the  Senate  Committee  on  Governmental  Affairs: 

The  inability  to  obtain  sufficient  information  in  debriefing,  in  particular  a 
meaningful  explanation  of  the  basis  for  the  award  decision,  leads  many  firms  to 
file  protests  in  order  to  obtain  that  information.  The  primary  purpose  of 
mandating  meaningful  debriefings  as  part  of  the  acquisition  streamlining  effort  is 
to  eliminate  the  filing  of  protests  as  a  means  to  discover  the  propriety  of  an  award 
decision.  (Allen  and  others,  1997:  25) 
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Post-award  debriefings  are  not  appropriate  for  all  contract  awards.  They  apply  to 
negotiated  procurements  only.  In  the  federal  government,  there  are  two  basic  methods  of 
acquiring  goods  and  services:  sealed  bidding  and  competitive  negotiations.  Sealed 
bidding  (previously  called  formal  advertising)  is  the  process  of  selecting  contractors  on 
the  basis  of  price  alone  through  publicly  opened  bids  (Cibinic  and  Nash,  1986:  387). 
Competitive  negotiation  procedures,  on  the  other  hand,  are  much  less  formal  but  much 
more  complex.  Contract  award  under  competitive  negotiations  is  not  necessarily  made  to 
the  lowest  offeror.  Government  procurement  personnel  have  much  greater  discretion  in 
source  selection  and  often  “trade  off’  price  against  other  factors,  such  as  technical  merit 
or  managerial  competence,  when  deciding  who  gets  the  award  (Cibinic  and  Nash,  1989: 
524).  As  a  result,  unsuccessful  offerors  may  not  have  a  full  understanding  of  the  basis  for 
the  award,  and  may  therefore  request  a  post-award  debriefing  in  order  to  gain  insight  into 
the  agency  source  selection  decision. 

Unsuccessful  offerors  do  not  always  request  a  post-award  debriefing,  however,  or, 
if  they  do,  they  may  not  get  all  the  information  they  want.  As  a  consequence,  some 
unsuccessful  offerors  may  file  a  protest  in  an  attempt  to  get  the  award  overturned  in  their 
favor.  Unsuccessful  offerors  have  many  forums  in  which  to  lodge  protests:  the  General 
Accounting  Office  (GAO),  a  federal  district  court,  the  Court  of  Federal  Claims,  or  the 
awarding  agency  itself  (Fausti,  1997:  10).  Protests  based  on  improprieties  in  the 
solicitation  must  be  filed  “prior  to  bid  opening  or  the  due  date  for  receipt  of  proposals” 
(Federal  Contracts  Report,  1995:  2).  Most  other  protests  must  be  filed  within  14  calendar 
days  of  “the  date  when  the  basis  of  the  protest  is  known  or  should  have  been  known, 
whichever  is  earlier”  (Federal  Contracts  Report,  1995: 2).  Once  a  protest  has  been  filed. 
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agencies  must  ordinarily  suspend  contract  award  or  stop  work  on  awarded  contracts.  In 
post-award  protests,  agencies  can  continue  performance  if  “urgent  and  compelling 
circumstances”  warrant  (Fausti,  1997:  12).  In  any  case,  protests  are  a  burden  to  both  the 
protester  and  the  agency  in  terms  legal  and  administrative  expenses. 

The  ABA  has  confirmed  that  some  protests  would  not  have  been  filed  if  a 
meaningful  post-award  debriefing  had  been  held  (The  Protest  Experience,  1989;  79). 
Before  the  Federal  Acquisition  Streamlining  Act  of  1994  (FAS A),  government 
contracting  officers  followed  the  strict  guidance  for  debriefing  unsuccessful  offerors 
contained  in  Part  15  of  the  Federal  Acquisition  Regulation.  The  general  belief 
concerning  these  debriefings  was  that  they  were  pro  forma  and  provided  little  information 
of  real  value  to  the  disappointed  bidder.  One  of  the  tenets  of  acquisition  reform  was  that 
disappointed  bidders  should  have  reasonable  access  to  the  reasons  for  agency  decisions  in 
source  selection  (Section  800  Report,  1993:  42).  The  belief  was  that  precipitous  protests 
could  be  avoided  by  providing  unsuccessful  offerors  more  complete  and  timely 
information  on  source  selection  decisions.  This  would,  in  turn,  shorten  the  procurement 
cycle  and  save  time  and  expense  for  both  parties. 

Research  Objective 

The  primary  objective  of  this  research  is  to  determine  whether  FAS  A  changes  in 
the  post-award  debriefing  process  have  resulted  in  fewer  protests.  There  are  other  aspects 
of  the  post-award  debriefing  process  that  also  warrant  investigation.  Some  view  the  post¬ 
award  debriefing  as  a  tool  to  help  contractors  prepare  better  proposals.  Another  research 
objective,  then,  to  determine  if  the  debriefing  process  has  motivated  contractors  to 
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produce  better,  more  competitive  proposals.  The  research  will  also  involve  a  comparison 
of  the  post-award  debriefing  proeess  before  and  after  FAS  A  changes  in  an  attempt  to 
document  the  evolution  of  the  process  and  the  effectiveness  of  the  FASA  changes. 

Contractors  having  experience  with  both  pre-  and  post-FASA  debriefings  will  be 
interviewed  to  gain  their  perspective  on  the  changes  in  the  process  and  to  help  determine 
whether  the  changes  have  had  their  intended  effects.  The  research  will  be  augmented  by 
an  examination  of  government  post-award  debriefing  documentation  and  interviews  with 
key  government  personnel.  The  research  will  be  conducted  at  Air  Force  Materiel 
Command,  Aeronautical  Systems  Center  (ASC). 

Investigative  Questions 

The  following  investigative  questions  have  been  developed  to  guide  this  research: 

(1)  How  does  the  new  post-award  debriefing  process  differ  from  the  old,  pre- 
FASA  process? 

(2)  How  do  government  and  contractor  personnel  view  the  differences  between 
the  old  and  new  debriefing  processes? 

(3)  Has  the  new,  expanded  post-award  debriefing  process  helped  contractors  to 
develop  better  proposals  for  future  procurements? 

(4)  What  has  been  the  impact  of  the  new  debriefing  process  on  the  likelihood  of  a 
debriefed  offeror’s  pursuing  a  protest? 

(5)  What,  if  anything,  can  be  done  to  improve  the  debriefing  process? 


5 


Scope  and  Limitations 


This  research  will  focus  on  the  post-award  debriefing  process  at  Air  Force 
Materiel  Command,  Aeronautical  Systems  Center  (ASC).  It  will  document  the  changes 
in  the  process  as  a  result  of  the  Federal  Acquisition  Streamlining  Act  of  1994  and  will 
attempt  to  determine  the  effects  of  those  changes  in  such  areas  as  proposal  quality  and 
likelihood  of  protest. 

This  study  will  not  involve  quantitative  analysis.  The  purpose  is  not  to  provide  a 
broad  statistical  analysis  of  the  numbers  of  pre-  and  post-FASA  protest  actions.  Instead, 
this  study  is  a  qualitative  study  based  on  interviews  with  government  and  industry 
personnel  and  on  analysis  of  government  file  documentation.  A  primary  purpose  is  to 
determine  if  the  expanded  debriefing  process  has  reduced  the  likelihood  protest.  Care 
will  be  taken  to  assure  a  reasonable  degree  of  reliability  and  validity  of  the  findings,  but 
broad,  generalizable  conclusions  are  not  necessarily  the  goal.  Perhaps  this  research  can 
serve  as  the  basis  for  a  future  researcher  to  develop  a  quantitative  study  of  the 
effectiveness  of  the  post-award  debriefing  process. 

Summary 

This  chapter  described  the  general  issues  concerning  bid  protest  and  the  post¬ 
award  debriefing  process.  It  provided  background  in  which  to  frame  the  research 
problem  and  stated  the  research  objective,  investigative  questions,  and  scope  and 
limitations  of  the  study.  The  next  chapter  will  examine  the  literature  describing  bid 
protests  and  debriefings,  and  the  laws  and  regulations  that  have  affected  these  processes. 
Chapter  III,  Methodology,  will  explain  in  detail  the  data  collection  and  analysis 
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techniques  employed  in  this  research.  Chapter  IV  will  include  an  analysis  of  the  data  and 
Chapter  V  will  draw  conclusions  and  suggest  areas  for  future  study. 
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II.  Literature  Review 


Introduction 

This  chapter  provides  an  overview  of  the  bid  protest  and  post-award  debriefing 
processes,  and  a  review  of  the  literature  in  these  areas.  It  begins  by  defining  bid  protest 
and  discussing  the  laws  and  regulations  that  affect  the  bid  protest  process,  including  the 
Truth  in  Negotiations  Act,  the  Federal  Acquisition  Streamlining  Act  of  1994,  and  the 
Federal  Acquisition  Regulation.  The  post-award  debriefing  process  is  then  defined  and 
discussed,  with  a  focus  on  the  historical  problems  that  FASA  was  designed,  in  part,  to 
eliminate.  Next  comes  a  look  at  why  contractors  protest,  with  an  important  historical 
reason  being  the  lack  of  communication  during  the  post-award  debrief.  The  FASA 
changes  affecting  post-award  debriefings  are  then  discussed  and  examined.  The  chapter 
concludes  with  a  review  of  past  research  into  the  post-award  debriefing  process. 

What  is  a  Protest? 

Questions  often  arise  concerning  the  government’s  actions  in  the  course  of 
awarding  a  contract.  A  procurement  protest  (or  “bid  protest”)  occms  when  a  bidder  or 
offeror  formally  objects  to  procurement  actions  or  decisions  by  government  personnel. 
The  Federal  Acquisition  Regulation  (FAR)  defines  a  protest  as  “a  written  objection  by  an 
interested  party  to  any  of  the  following: 

(a)  A  solicitation  or  other  request  by  an  agency  for  offers  for  a  contract  for  the 
procurement  of  property  or  services. 

(b)  The  cancellation  of  the  solicitation  or  other  request. 
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(c)  An  award  or  proposed  award  of  the  contract. 

(d)  A  termination  or  cancellation  of  an  award  of  the  contract,  if  the  written 
objection  contains  an  allegation  that  the  termination  or  cancellation  is  based  in  whole  or 
in  part  on  improprieties  concerning  the  award  of  the  contract.”  (FAR  33.101). 

An  “interested  party”  refers  to  an  actual  or  prospective  bidder  with  a  direct 
economic  interest  in  the  procurement  (FAR  33.101).  The  General  Accoimting  Office 
says  that  an  interested  party  is  normally  the  party  that  would  be  in  line  for  award  if  the 
protest  were  “sustained,”  or  granted  (Bid  Protests  at  GAO:  Descriptive  Guide,  1996:  4). 

The  protest  serves  as  a  means  of  ensuring  that  the  government  conducts 
procurements  in  accordance  with  statutory  and  regulatory  procedures,  from  establishment 
of  requirements  and  publication  of  a  solicitation  through  proposal  evaluation,  negotiation, 
and  contract  award.  Congress  has  recognized  the  important  role  of  bid  protests  in  helping 
to  promote  integrity  and  competition  in  government  procurement.  In  adopting  the 
Competition  in  Contracting  Act  in  1984,  Congress  stated  that  formal  protest  proceedings 
were  essential  because: 

...a  strong  enforcement  mechanism  is  necessary  to  ensure  the  mandate  for 
competition  is  enforced  and  that  vendors  wrongly  excluded  from  competing  for 
government  contracts  receive  equitable  relief.  (Section  800  Report,  1993:  40) 

The  numerous  laws  and  regulations  that  govern  the  federal  procurement  process 

are  designed  to  ensure  that  procurements  by  the  federal  government  are  conducted  fairly, 

honestly,  and  with  the  maxim  amount  of  competition  possible  under  the  circumstances. 

Two  laws  that  have  had  a  major  impact  on  the  procurement  process,  including  the  protest 

process,  were  the  Competition  in  Contracting  Act  of  1984  (CICA)  and  the  Federal 
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Acquisition  Streamlining  Act  of  1994  (FASA).  The  impact  of  both  acts  on  the  protest 
process  in  particular  will  be  considered  in  turn. 

The  Competition  in  Contracting  Act 

The  enactment  of  the  Competition  in  Contracting  Act  of  1984  brought  major 
changes  in  the  protest  process.  The  CICA  requirement  for  “full  and  open  competition” 
became  a  critical  factor  in  awarding  government  contracts.  To  enforce  the  CICA 
mandate  for  competition  and  other  procurement  requirements,  Congress  for  the  first  time 
provided  express  statutory  authority  for  the  General  Accounting  Office  (GAO)  to  review 
bid  protests.  In  addition  to  providing  statutory  authority,  Congress  also  made  other  major 
changes  to  the  GAO  bid  protest  process  itself.  First,  CICA  imposed  a  specific  time  limit 
of  ninety  days  for  GAO  to  render  a  decision.  Second,  CICA  stipulated  that  an  agency 
may  not  award  a  contract— or  must  suspend  contract  performance  until  the  protest  is 
resolved--if  the  protest  was  filed  before  award  or  within  ten  calendar  days  after  award. 
(This  so-called  “stay”  provision  could  be  lifted  if  the  contracting  officer  determined  in 
writing  that  one  of  three  exceptions  applied.)  Third,  CICA  authorized  the  GAO  to 
declare  an  “interested  party”  to  be  entitled  to  recover  protest  costs,  including  bid  or 
proposal  preparation  costs  and  reasonable  attorney  fees.  CICA  defined  an  interested 
party  as  “an  actual  or  prospective  bidder  or  offeror  whose  direct  economic  interest  would 
be  affected  by  the  award  of  the  contract  or  by  failure  to  award  the  contract.”  Last,  CICA 
established  a  sixty-day  maximum  time  for  agencies  to  implement  a  GAO 
recommendation,  or  report  to  the  GAO  why  the  recommendation  was  not  implemented 
(The  Protest  Experience,  1989:  4-5). 
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While  CICA  for  the  first  time  provided  a  statutory  basis  for  the  GAO  to  decide 
protests,  there  were  other  protest-related  provisions  of  CICA  that  generated  some 
controversy.  One  was  the  stay  provision  referred  to  above.  Under  CICA,  a  protest  to  the 
GAO  could  stop  contracting  agencies  from  awarding  the  contract,  or  from  continuing 
contract  performance  if  the  contract  had  already  been  awarded.  Agencies  could  override 
a  pre-award  suspension  upon  a  written  finding  that  “urgent  and  compelling 
circumstances”  would  not  permit  waiting  for  the  GAO’s  decision.  Post  award  stays  could 
be  lifted  if  the  agencies  made  a  determination  that  continued  performance  of  the  contract 
was  “in  the  best  interests  of  the  United  States”  (Pachter,  1985:  4).  Once  an  agency 
determination  had  been  made,  the  GAO  was  not  authorized  to  overturn  it. 

Another  controversial  aspect  of  CICA  with  respect  to  bid  protests  was  the 
provision  allowing  the  GAO  to  order  an  agency  to  reimburse  a  disappointed  bidder’s 
attorney’s  fees,  protest  costs,  and  bid  and  proposal  (B&P)  costs,  if  the  GAO  determined 
that  the  bidder  had  been  unreasonably  excluded  from  the  procurement.  The  agency  was 
to  pay  the  costs  out  of  its  current  account  for  the  proeurement  of  goods  and  services. 
(Fiedelman,  1996:  68). 

CICA  had  been  enacted  because  Congress  had  perceived  that  federal  agencies 
were  not  doing  a  good  job  of  exploiting  the  competitive  forees  of  the  private  sector. 

The  intent  of  CICA  was  to  promote  full  and  open  competition  by  relying  in  part  on 
private  enforcement  of  procurement  laws  and  regulations.  Competition  statistics  indicate 
that  CICA  had  its  intended  effect.  In  FY  1983,  the  year  before  CICA  became  law,  40 
percent  of  federal  procurement  dollars  were  competitively  awarded.  By  FY  1993,  nine 
years  after  CICA,  almost  70  percent  of  all  procurements  stemmed  from  competition 
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(Pegnato,  1995:  69).  Not  intended,  however,  was  the  proliferation  of  procurements 
challenged  by  protests.  The  data  show  that  in  the  ten-year  period  from  1984  through 
1993,  the  number  of  protests  increased  by  77  percent  (Pegnato,  1995:  71).  Many  of  these 
protests  were  considered  by  both  sides  to  be  frivolous.  In  the  years  immediately 
following  the  passage  of  CICA,  protest  proliferation  was  referred  to  as  the  “twenty-two 
cent  injunction,”  where  any  disappointed  bidder  with  a  twenty-two  cent  stamp  could  stop 
an  agency  from  awarding  a  contract,  or  could  bring  contract  performance  to  a  halt 
(Pachter,  1985:  4).  Federal  agencies  complained  about  the  expense  and  disruption  that 
these  protests  were  inflicting  upon  the  government  contracting  community.  It  appeared 
that,  in  an  attempt  to  stop  perceived  agency  abuses  of  the  procurement  process.  Congress 
had  now  created  a  situation  where  contractors  had  become  a  major  source  of  the  abuse 
(Gabig,  1988:  22). 

The  Federal  Acquisition  Streamlining  Act 

The  Federal  Acquisition  Streamlining  Act  of  1994  is  considered  by  many  to  be 
the  most  sweeping  legislation  in  federal  government  contracting  since  the  passage  of 
CICA.  FASA  affected  broad  range  acquisition  and  procurement  issues.  The  changes  in 
FASA  were  designed  to  reflect  a  fundamental  shift  toward  the  application  of  commercial 
practices  to  public  sector  procurement.  Based  largely  on  the  recommendations  contained 
in  the  Report  of  the  Acquisition  Law  Advisory  Panel  to  the  United  States  Congress  (the 
“Section  800  Panel”),  FASA  repealed,  or  substantially  modified,  over  225  provisions  of 
law  affecting  the  federal  procurement  process  (Lumer,  1994:  1). 
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Like  CICA,  FASA  also  had  a  major  impact  on  the  bid  protest  process.  For 
example,  FASA  now  requires  agencies  to  suspend  contract  performance  if  the  agency 
receives  a  protest  notice  from  the  GAO  either  ten  calendar  days  after  contract  award,  or 
five  calendar  days  after  the  “offered”  date  of  a  “required”  debriefing,  whichever  is  later 
(Allen  and  others,  1997:  89).  Perhaps  the  biggest  impact,  however,  has  been  in  the  area 
of  post-award  debriefing  of  unsuccessful  offerors.  Under  competitive  negotiation 
procedures,  contract  award  is  often  determined  by  factors  other  than  price  alone.  There  is 
no  public  bid  opening  as  in  sealed  bidding,  and  there  is  usually  a  lengthy  period  of 
proposal  evaluation  and  negotiation  before  award  is  made.  Given  that  award  is  not 
always  based  solely  on  price,  unsuccessful  offerors  may  not  understand  fully  why  they 
did  not  receive  the  award.  The  reason  may  not  be  simply  that  the  price  was  too  high,  but 
that  technical,  management,  and/or  other  factors  may  have  been  deficient.  Only  by 
requesting  a  post-award  debriefing  from  the  contracting  officer  can  the  unsuccessful 
offerors  learn  the  reasons  why  their  proposals  were  not  selected.  (Cooper,  1995:3-4) 

What  is  a  Post-Award  Debriefing? 

A  post-award  debriefing  is  a  meeting  between  the  contracting  officer  and  an 
unsuccessful  offeror  after  a  contract  has  been  awarded.  According  to  FAR  15.1006,  the 
contracting  officer  must  provide  a  debriefing  if  he  or  she  receives  a  written  request  from 
an  unsuccessful  offeror  within  three  days  of  the  notice  of  contract  award.  The  U.  S. 

Army  Debriefing  Handbook  lists  the  following  purposes  of  a  debriefing: 

(a)  To  explain  the  rationale  for  the  contract  award  decision. 

(b)  To  assure  the  unsuccessful  offeror  that  it  received  fair  treatment. 
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(c)  To  assure  the  offeror  that  all  proposals  were  evaluated  in  accordance  with  the 
solicitation  and  applicable  laws  and  regulations. 

(d)  To  identify  weaknesses  in  the  offeror’s  proposal  so  that  the  offeror  can  prepare 
better  proposals  in  the  future. 

(e)  To  reduce  misunderstandings  and  protests.  (AMC  Pamphlet  715-3:  2) 

In  addition,  FAR  15.1006(e)  makes  it  clear  that  debriefings  do  not  involve  point-by-point 
comparisons  with  other  offerors’  proposals,  nor  do  they  involve  the  release  of  trade 
secrets  or  other  privileged  or  confidential  information. 


Debriefing  Problems 


CICA  did  not  disturb  the  long-standing  FAR  requirement  that  contracting  officers 
provide,  upon  written  request  from  an  unsuccessful  offeror,  a  post-award  debriefing. 
Debriefings  were  intended  to  give  contractors  insight  into  the  proposal  evaluation  process 
so  that  future  proposals  could  be  more  competitive.  This  concept  of  using  the  debriefing 
as  a  tool  to  promote  better  proposals  was  recognized  in  Motorola,  Inc.,  where  the 
Comptroller  General  stated  that  the  “function  of  a  debriefing  is  not  to  justify  or  defend  a 
selection  decision  but  to  assist  offerors  in  preparing  future  proposals”  {Motorola,  1989). 

In  practice,  however,  debriefings  had  an  almost  opposite  effect,  as  the  following 
quote  from  Rand  L.  Allen  and  others  in  the  Acquisition  Streeimlining  Institute  Course 
Manual  illustrates: 

For  many  years,  offerors  have  complained  generally  that  debriefings  were  not 
serving  their  intended  purpose.  Ideally,  a  debriefing  should  inform  an  offeror  why 
it  lost  the  competition,  in  order  to  improve  that  company’s  proposals  in 
subsequent  procurements  and  thereby  further  full  and  open  competition. 

However,  Government  personnel  in  debriefings  increasingly  had  become  less 
willing  to  divulge  significant  information  about  the  procurements,  usually  guided 
by  the  fear  of  revealing  proprietary  or  source  selection  sensitive  information. 
Companies  had  been  left  with  little  option  other  than  bid  protests  to  discover  the 
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basis  of  an  award  decision.  This  became  a  self-fulfilling  prophecy.  (Allen  and 
others:  1997:  27) 

The  problem  with  debriefings  in  the  days  before  FASA,  then,  was  that  they 

provided  little  or  no  useful  information  to  the  disappointed  bidder.  The  entire  FAR 

guidance  on  debriefings  was  contained  in  one  short  paragraph  under  FAR  15.1003(b): 

Debriefing  information  shall  include  the  Government’s  evaluation  of  the 
significant  weak  or  deficient  factors  in  the  proposal;  however,  point-by-point 
comparisons  with  other  offerors’  proposals  shall  not  be  made.  Debriefing  shall 
not  reveal  the  relative  merits  or  technical  standing  of  competitors  or  the  evaluation 
scoring.  Moreover,  debriefing  shall  not  reveal  any  information  not  releasable 
under  the  Freedom  of  Information  Act. . . . 

The  FAR  provided  more  information  concerning  what  not  to  say  in  a  debriefing, 
rather  than  what  to  say.  Consequently,  as  Allen  claims,  contracting  officers  were 
reluctant  to  be  open  and  frank,  fearing  that  the  more  information  they  gave,  the  better  the 
chance  of  violating  a  regulation  or  of  divulging  sensitive  or  proprietary  information  and 
triggering  a  protest.  Furthermore,  Cooper  points  out  that  the  effects  of  procurement 
legislation  often  put  contracting  officers  between  a  rock  and  a  hard  place.  For  example, 
CICA  required  contracting  officers  to  maintain  open  communication  channels  with 
industry  in  order  to  understand  the  capabilities  of  the  private  sector  and  to  promote  full 
and  open  competition.  On  the  other  hand,  procurement  integrity  laws  such  as  the  Ethics 
Reform  Act  of  1989  forced  contracting  officers  to  restrict  the  flow  of  sensitive 
procurement  information  to  maintain  the  integrity  of  the  procurement  process.  As 
Cooper  notes,  the  steep  penalties  for  violating  the  procurement  integrity  laws  had  a 
“chilling  effect  on  the  flow  of  information  between  government  and  industry”  (Cooper, 
1995:  2).  Under  such  conditions,  it  is  not  surprising  that  contracting  officers  often  erred 
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on  the  side  of  caution  in  debriefings  by  choosing  not  to  divulge  much  specific 
information. 

Why  Do  Unsuccessful  Offerors  Protest? 

There  are  many  reasons  why  unsuccessful  offerors  protest.  While  most  protests 
arise  over  objections  to  the  rejection  of  a  bid  or  proposal,  other  reasons  for  protest  include 
alleged  restrictive  or  defective  specifications,  omission  of  a  required  provision,  or 
ambiguous  evaluation  criteria  in  the  solicitation  (Bid  Protests  at  GAO:  A  Descriptive 
Guide,  1996:  4).  The  cancellation  of  a  solicitation  or  termination  of  a  contract  can  also 
precipitate  a  protest. 

Lieberman  has  categorized  and  documented  the  results  of  protests  filed  in 
different  forums  over  a  three-year  period.  He  grouped  protests  into  the  following  major 
categories  and  sub-categories  based  on  greatest  likelihood  of  the  protest’s  being 
sustained: 

(1)  Improper  actions  in  evaluation  and  negotiation  of  proposals 

(a)  Failure  to  follow  evaluation  criteria  in  the  solicitation 

(b)  Failure  to  conduct  discussions  properly 

(c)  Unreasonable  cost  realism  analysis 

(d)  Improper  exclusion  from  competitive  range 

(e)  Improper  cost/technical  tradeoff 

(f)  Improper  award  based  on  initial  proposals  without  discussion 

(2)  Improper  actions  involving  solicitations  or  requirements 

(a)  Defect  in  solicitation 
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(b)  Improper  action  involving  change  in  requirements  or  amendment  or 
cancellation  of  solicitation 

(c)  Failure  to  provide  bidder  with  amendment  to  solicitation 

(3)  Improper  restrictions  on  competition 

(a)  Improper  restrictions  on  competition — general 

(b)  Improper  sole  source  award 

Other  categories  included  improper  actions  involving  small  business  awards  and 
improper  acceptance  of  nonconforming  proposals  or  nonresponsive  bids  (Lieberman, 
1995:  11-19). 

Lieberman  categorized  protest  results  based  on  greatest  likelihood  of  the 
protester’s  receiving  relief,  but  the  overall  sustainment  rate  is  quite  low.  The  GAO,  for 
instance,  grants  relief  in  only  12-13  percent  of  the  protests  it  hears  (Lieberman,  1995:  23). 
Many  cases,  however,  are  settled,  dismissed,  or  withdrawn  before  a  decision  is  reached. 
Lieberman  cites  the  Protester  Effectiveness  Rate  (PER),  used  by  GAO,  that  includes  these 
cases  in  the  overall  sustainment  rate.  According  to  Lieberman,  the  PER  has  averaged 
29.8%  over  the  ten-year  period  from  1985  through  1994  (Lieberman,  1995:  5-6).  Even 
so,  given  that  the  total  number  of  protests  received  by  GAO  exceeds  2,500  annually 
(Metzger  and  Golden,  1997:  viii)  the  number  of  adjudicated  cases  is  still  quite  high.  This 
number  represents  a  great  cost  to  both  government  and  industry,  and  underscores  the  fact 
that  government  agencies  must  pay  special  attention  to  those  areas  most  likely  to 
precipitate  a  protest. 

While  Lieberman  lists  the  “official”  reasons  for  protest  by  unsuccessful  offerors, 
there  are  probably  many  other  unstated  factors  that  play  a  role  in  a  contractor’s  decision 
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to  protest,  such  as  financial  health  of  the  firm,  the  size  of  the  defense  budget,  political 
considerations  within  DOD,  overall  economic  conditions,  etc.  Cooper  interviewed  a 
defense  industry  executive  who  posited  two  underlying  reasons  for  filing  protests:  (1)  to 
gain  the  contract  (presumably  through  legal  maneuvering),  or  (2)  to  correct  perceived 
inequities.  He  says  that  perceived  inequities  usually  result  from  poor  communication  at 
the  post-award  debriefing.  Cooper  quotes  the  executive,  “In  many  cases,  we  discover  that 
what  we  perceived  as  an  unfair  practice  was  in  fact  good  business  poorly  communicated” 
(Cooper,  1995:  5). 

The  American  Bar  Association  stresses  the  importance  of  communication  in 
avoiding  protests.  It  cites  as  a  critical  factor  in  the  decision  to  protest  the  availability  of 
timely  and  accurate  information:  “To  the  extent  a  protester  can  obtain  timely  information 
concerning  whether  it  is  in  line  for  award  and  can  establish  that  its  offer  was  treated 
fairly,  protests  may  be  avoided”  (The  Protest  Experience,  1989:  79). 

Government  “Protest  Avoidance” 

Nash  and  Cibinic  argue  that  some  agencies  base  their  procurement  actions  more 
on  protest  avoidance  strategies  than  on  good  business  practices  (Nash  and  Cibinic, 
1994:33).  As  Nash  and  Cibinic  point  out,  such  strategies  rarely  work  and  often  end  up 
costing  the  agencies  more  in  the  long  run.  They  recommend  that  contracting  agencies, 
rather  than  being  guided  by  a  fear  of  protests,  should  put  the  government’s  best  interests 
first.  They  suggest  that  agency  representatives  be  as  open  and  forthcoming  as  possible  in 
discussions  and  debriefings:  “Many  protests  are  based  upon  suspicion  engendered  by  a 
failure  of  the  agency  to  respond  fi'ankly  to  the  protester’s  concern”  (Nash  and  Cibinic, 
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1994:  33).  Nash  and  Cibinic  believe  that  taking  time  to  answer  the  protester’s  questions 
and  to  fully  explain  the  procurement  rules  and  regulations  “will  not  do  away  with  protests 
but  will  undoubtedly  eliminate  some”  (Nash  and  Cibinic,  1994:  33). 

FAS  A  Expands  Scone  of  Debriefing 

The  theory  behind  post-award  debriefing  is  that  communication  between  the 
government  and  the  unsuccessful  offeror  should  reduce  the  number  of  protests.  FASA 
emphasizes  the  role  that  communication  between  government  and  industry  plays  in 
helping  to  avoid  protests.  Under  FASA,  the  scope  of  the  post-award  debriefing  was 
expanded.  The  government  must  now  provide  more  information  to  unsuccessful  offerors. 
In  the  course  manual  on  acquisition  streamlining  by  Federal  Publications,  Inc.,  Allen  and 
others  state  the  following: 

The  new  debriefing  rules  require  the  Government  to  disclose  much  more 
information  not  only  on  how  the  Government  evaluators  viewed  proposals  but 
also  on  the  evaluation  process  itself  The  intent  of  the  new  rules  is  to  provide 
businesses  information  to  help  them  do  better  in  the  future  and  also  to  stave  off 
future  protests.  (Allen  and  others,  1997:  15) 

According  to  FAR  15.1006(d),  debriefing  information  shall  include  at  a 
minimum: 

(1)  The  Government’s  evaluation  of  the  significant  weaknesses  or  deficiencies  in 
the  offeror’s  proposal,  if  applicable; 

(2)  The  overall  evaluated  cost  or  price  and  technical  rating,  if  applicable,  of  the 
successful  offeror  and  the  debriefed  offeror; 

(3)  The  overall  ranking  of  all  offerors  when  any  ranking  was  developed  by  the 
agency  during  the  source  selection; 

(4)  A  summary  of  the  rationale  for  award; 
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(5)  For  acquisitions  of  commercial  end  items,  the  make  and  model  of  the  item  to 
be  delivered  by  the  successful  offeror;  and 

(6)  Reasonable  responses  to  relevant  questions  about  whether  source  selection 
procedures  contained  in  the  solicitation,  applicable  regulations,  and  other 
applicable  authorities  were  followed. 

The  FASA  changes  are  significant  for  two  important  reasons.  First,  the 
government  must  present  the  overall  evaluated  cost  and  technical  ratings  of  both  the 
successful  offeror  and  the  debriefed  offeror.  In  the  past,  only  the  unsuccessful  offeror’s 
technical  ranking  and  cost  evaluation  was  provided.  Under  the  expanded  debriefing,  the 
successful  offeror’s  technical  and  cost  evaluations  are  also  provided.  The  offeror  being 
debriefed  now  has  a  basis  for  comparing  its  proposal  with  the  winning  proposal 
(Pickarz,  1995:  3). 

The  second  important  change  focuses  on  the  source  selection  decision  itself 
Previously,  in  explaining  their  source  selection  decisions,  contracting  officers  simply 
provided  general  statements  to  the  effect  that  the  winning  proposal  offered  the  overall 
best  value  in  meeting  the  government’s  requirements.  Such  statements  offered 
unsuccessful  offerors  little,  if  any,  insight  into  the  strengths  and  weaknesses  of  their 
proposals.  The  new  rules  force  contracting  officers  to  be  more  specific  in  explaining  why 
the  winning  proposal  was  selected. 

Pickarz  suggests  that  the  best  way  to  explain  the  source  selection  decision  is  to 
make  the  source  selection  decision  document  available  to  the  unsuccessful  offeror  at  the 
post-award  debriefing  (Pickarz,  1995:3).  Although  the  FAR  does  not  specifically  define 
“source  selection  decision  document,”  FAR  15.612(d)(2)  contains  the  following 
guidance: 
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The  supporting  documentation  prepared  for  the  selection  decision  shall  show  the 

relative  strengths,  weaknesses,  and  risks  in  terms  of  the  evaluation  factors.  The 

supporting  documentation  shall  include  the  basis  and  reasons  for  the  decision. 

Nash  and  Cibinic  point  out  that  almost  all  agencies  require  some  type  of  formal 
documentation  supporting  the  source  selection  decision  (Nash  and  Cibinic,  1993:  507). 
They  state  that  guidance  on  the  content  and  format  of  the  documentation  is  limited,  but 
cite  the  guidance  in  a  General  Services  Administration  (GSA)  Handbook,  Source 
Selection  Procedures,  as  being  typical: 

The  selection  statement  should  stand  alone  and  cover  the  following  basic  points; 

(1)  A  description  of  the  acquisition. 

(2)  The  names  of  the  offerors. 

(3)  The  competitive  ranking  of  the  proposals. 

(4)  A  summation  of  the  strengths  and  weaknesses  of  each  proposal  and  offeror. 

(5)  Reasons  why  the  firm  selected  provides  the  greatest  probability  of  satisfying 

the  Government’s  requirements.  (Nash  and  Cibinic,  1993:  507) 

By  providing  the  source  selection  decision  document  to  the  unsuccessful  offeror 
at  the  post-award  debriefing,  the  contracting  officer  will  answer  the  obvious  question  of 
why  the  winning  contractor  was  selected.  According  to  Pickarz,  however,  another 
purpose  is  also  served:  maintaining  good  relations  with  the  supplier  base.  Because  of  the 
intense  time,  effort,  and  expense  required  in  proposal  preparation  and  negotiation,  losing 
contractors  must  not  be  left  feeling  that  they  were  treated  unfairly  and  unprofessionally. 
Taking  care  not  to  divulge  proprietary  information,  contracting  officers  must  be  as  fair 
and  open  as  possible  in  the  post-award  debriefing  (Pickarz,  1995: 4). 
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Previous  Research  and  Focus  of  Current  Study 


Although  much  has  been  written  about  the  debriefing  process,  there  has  been  very 
little  actual  research  in  the  area  of  post-award  debriefings.  Two  recent  master’s  theses, 
summarized  below,  have  researched  the  debriefing  process  from  similar  perspectives 
within  different  defense  agencies. 

In  1994,  Nutbrown  reviewed  Army  regulations  covering  debriefings,  investigated 
the  then  current  debriefing  process,  and  recommended  steps  to  improve  the  process 
within  Army  Material  Command.  He  recommended,  among  other  things,  that  debriefings 
be  held  in  a  timely  fashion,  that  all  government  evaluators  attend  debriefings  (  not  just 
contracting  officers),  and  that  the  source  selection  decision  document  be  made  available 
(Nutbrown,  1994:  125).  His  research  was  conducted  largely  before  the  full 
implementation  of  FASA. 

Cooper’s  research  into  the  debriefing  process  a  year  later  included  FASA  changes. 
He  examined  the  process  from  the  standpoint  of  a  Navy  installation  contracting  officer. 
Among  his  recommendations  were  to  release  all  information  an  unsuccessful  offeror 
would  normally  receive  under  protest  discovery  procedures  and  to  implement  a  training 
program  for  conducting  post-award  debriefings  (Cooper,  1995:  55-56). 

Neither  of  the  above  papers  focused  on  the  main  subject  of  this  research: 
the  effect  of  acquisition  reform  on  the  post-award  debriefing  process.  Has  the  new 
process  resulted  in  more  competitive  proposals  from  industry?  Has  it  decreased  the 
likelihood  of  protest?  Finding  answers  to  these  questions  and  other  questions  related  to 
the  new  post-award  debriefing  process  is  the  primary  focus  of  this  research  effort. 
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Summary 


This  chapter  reviewed  the  literature  on  protests  in  general  and  the  post-award 
debriefing  process  in  particular.  It  provided  background  information  to  place  the  study 
within  the  appropriate  context.  The  protest  and  post-award  debriefing  processes  were 
defined  and  described  in  detail.  In  addition,  the  impact  of  important  legislation  on  these 
processes  was  discussed,  along  with  an  analysis  of  the  changes  that  resulted  in  the 
expanded  post-award  debriefing  process.  The  discussion  included  insight  into  some  of 
the  problems  in  the  post-award  debriefing  process  that  specific  legislation  was  intended  to 
eliminate.  The  review  included  an  examination  of  the  reasons  why  vmsuccessful  offerors 
protest  agency  award  decisions,  and  also  discussed  the  emphasis  and  results  of  previous 
research  in  the  area  of  post-award  debriefing. 

The  literature  review  suggests  that  previous  research  into  post-award  debriefing 
has  been  process-oriented.  That  is,  it  has  focused  on  how  post-award  debriefings  have 
been  conducted.  Process  is  also  an  important  aspect  of  this  study,  but  the  primary  focus 
is  the  impact  of  the  expanded  post-award  debriefing  process  on  contractor  behavior. 

How  have  the  FASA  changes  affected  unsuccessful  offerors?  Has  the  impact  been 
positive?  Have  unsuccessful  offerors  gained  information  to  allow  them  to  develop  better, 
more  competitive  proposals  for  future  procurements?  Have  unsuccessful  offerors  decided 
not  to  protest  based  on  information  obtained  during  a  post-award  debriefing? 

The  existing  research  has  not  addressed  these  areas.  The  next  chapter  focuses  on 
the  research  methodology  that  vyill  be  employed  in  an  attempt  to  answer  such  questions. 
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III.  Methodology 


Introduction 

The  impetus  for  this  study  was  acquisition  reform,  specifically  the  Federal 
Acquisition  Streamlining  Act  of  1994  (FAS A),  which,  among  other  things,  directed 
changes  in  the  post-award  debriefing  process.  The  overall  purpose  of  the  research  was  to 
determine  whether  the  expanded  post-award  debriefing  process  plays  a  significant  role  in 
a  contractor’s  decision  not  to  protest.  As  discussed  in  the  literature  review,  this  research 
was  a  qualitative  study.  This  chapter  will  define  qualitative  research,  describe  qualitative 
research  “design,”  and  discuss  the  methodological  approach  for  the  current  study. 

What  is  Qualitative  Research? 

Strauss  and  Corbin  define  qualitative  research  as  “any  kind  of  research  that 
produces  findings  not  arrived  at  by  means  of  statistical  procedures  or  other  means  of 
quantification”  (Strauss  and  Corbin,  1990:  17).  They  stress  that  qualitative  research  is 
not  the  quantifying  of  qualitative  data  gathered  by  means  of  interview  or  observation. 
Qualitative  research  is,  rather,  a  “nonmathematical  analytical  procedure”  that  results  in 
findings  from  data  gathered  through  a  variety  of  means,  including  interview,  observation, 
document  analysis,  etc.  (Strauss  and  Corbin,  1990:  18). 

Creswell  says  that  qualitative  research  is  inductive,  building  on  abstractions, 
concepts,  hypotheses,  and  theories  (Creswell,  1994:  145).  He  adds  that  qualitative 
researchers  are  more  concerned  with  process  than  with  products,  and  that  they  put  great 
emphasis  on  discerning  meaning — ^how  people  make  sense  of  the  world  and  their 
experiences  (Creswell,  1994:  145).  Finally,  Creswell  believes  that,  because  of  the 
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primacy  of  process  and  meaning,  qualitative  research  is  by  necessity  descriptive  and 
interpretive — ^understanding  comes  about  through  words  or  pictures  (Creswell,  1994: 
145). 


Deciding  whether  certain  problems  are  more  suited  for  a  quantitative  or 
qualitative  approach  is  a  debatable  proposition,  according  to  Creswell  (Creswell,  1994; 
10).  He  points  out  that  the  nature  of  the  problem  and  the  amount  of  existing  information 
are  important  factors.  He  provides  the  following  guidelines  in  helping  to  decide  on  a 
particular  approach: 

For  quantitative  studies  the  problem  evolves  from  the  literature,  so  a  substantial 
body  of  literature  exists  on  which  the  researcher  can  build.  Variables  are  known, 
and  theories  may  exist  that  need  to  be  tested  and  verified.  For  qualitative  studies 
the  research  problem  needs  to  be  explored  because  little  information  exists  on  the 
topic.  The  variables  are  largely  unknown,  and  the  researcher  wants  to  focus  on 
the  context  that  may  shape  the  understanding  of  the  phenomenon  being  studied. 

In  many  qualitative  studies  a  theory  base  does  not  guide  the  study  because  those 
available  are  inadequate,  incomplete,  or  simply  missing.  (Creswell,  1994:  10) 

Marshall  and  Rossman  also  provide  some  guidelines  for  the  types  of  research  that 

are  especially  appropriate  for  qualitative  studies: 

•  research  that  cannot  be  done  experimentally  for  practical  or  ethical  reeisons 

•  research  that  delves  in  depth  into  complexities  and  processes 

•  research  for  which  relevant  variables  have  yet  to  be  identified 

•  research  that  seeks  to  explore  where  and  why  policy  and  practice  do  not  work 

•  research  on  unknown  societies  or  innovative  systems 

•  research  on  informal  or  unstructured  linkages  and  processes  in  organizations 
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•  research  on  real,  as  opposed  to  stated,  organizational  goals 
(Marshall  and  Rossman,  1989:  46). 

Two  areas  listed  above  seem  to  describe  in  general  terms  the  purpose  of  the 
current  study.  This  research  will  delve  into  “complexities  and  processes”  and  will 
examine  “policy  and  practice”  as  they  relate  to  post-award  debriefings.  Further  support 
for  the  qualitative  nature  of  this  study  comes  from  Michael  Quinn  Patton  in  his  book. 
Qualitative  Evaluation  and  Research  Methods.  Patton  says  the  qualitative  research 
approach  is  especially  appropriate  for  studies  that  focus  on  the  effects  of  changing 
policies  or  programs.  Qualitative  research  is  also  an  effective  process  to  use  at  major 
transition  points,  where  change  is  introduced  and  the  expected  outcome,  as  well  as  any 
unintended  consequences,  needs  to  be  monitored  (Patton,  1990:  12). 

Finally,  Bryman  illustrates  the  nature  of  qualitative  research  in  a  series  of 
questions:  What  prompted  the  phenomenon?  What  did  people  think  of  it?  How  did  they 
react  to  it?  How  did  it  develop?  What  was  the  outcome?  What  did  people  think  of  the 
outcome?  The  emphasis  is  on  process,  which  Bryman  describes  as  “the  unfolding  of 
events  in  time”  (Bryman,  1989:  137). 

The  above  analysis  indicates  that  a  qualitative  research  method  is  appropriate  for 
this  study.  The  literature  review  revealed  that,  while  the  post-award  debriefing  process 
itself  has  been  studied,  no  previous  research  has  been  done  to  determine  whether  the 
expanded  debriefing  process  has  influenced  the  likelihood  of  contractor  protest.  The 
implementation  of  FASA  has  produced  a  situation  of  changing  policies  within  DOD.  The 
impact  of  these  changes  is  of  primary  importance  to  DOD  managers  and  policy  makers. 
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In  the  case  of  changing  post-award  debriefing  policy  and  procedures,  qualitative  research 
is  the  proper  approach  for  an  in-depth  and  detailed  analysis  that  attempts  to  determine  the 
effects  of  policy  changes  on  contractor  behavior.  The  question  now  is,  what  is  the 
appropriate  qualitative  research  design? 

Qualitative  Research  Design 

According  to  Creswell,  one  of  the  characteristics  of  a  qualitative  research  problem 

is  that  the  concept  is  “immature”  because  of  a  lack  of  previous  research  (Creswell,  1994: 

146).  By  its  very  nature,  then,  qualitative  research  is  exploratory.  Therefore,  unlike  in 

the  world  of  quantitative  research,  there  is  no  consensus  on  the  precise  procedures  for 

data  collection  and  analysis  of  qualitative  research  (Creswell,  1994:  143). 

The  foregoing  is  not  to  imply  that  qualitative  research  is  whimsical  and 

haphazard,  but  rather  that  there  is  no  standard  approach.  Marshall  and  Rossman 

recognize  a  wide  variation  in  qualitative  research  methods: 

. .  .the  approaches  vary,  depending  on  how  intrusive  the  researcher  is  required  to 
be  in  the  gathering  of  data,  whether  these  data  document  nonverbal  or  verbal 
behaviour  or  both,  whether  it  is  appropriate  to  question  the  participants  as  to  how 
they  view  their  worlds,  and  how  the  data  can  be  fruitfully  analysed.  (Marshall  and 
Rossman,  1989:  10-11) 

Marshall  and  Rossman  believe  that  qualitative  research  design  must  be  flexible  enough  to 
allow  researchers  the  freedom  to  explore  but  structured  enough  to  convince  readers  that 
the  researchers  know  what  they  are  doing  (Marshall  and  Rossman,  1989: 45). 

Patton  says  there  are  no  “rigid  rules,”  no  “recipe  or  formula”  to  follow  when 
deciding  upon  a  research  method  (Patton,  1990:  13).  He  states  that  just  as  in  art  where 
there  is  “no  single,  ideal  standard,”  the  “art  of  evaluation  includes  creating  a  design  and 
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gathering  information  that  is  appropriate  for  a  specific  situation  and  particular  decision¬ 
making  context”  (Patton,  1990:  13).  Patton  echoes  Marshall  and  Rossman  when  he  says 
that  qualitative  designs  must  not  be  rigid  as  the  word  “design”  implies,  but  must  maintain 
openness  and  flexibility  to  permit  exploration  of  the  subject  under  study  (Patton,  1989: 
196). 

Choosing  a  qualitative  research  design,  then,  appears  to  involve  just  as  much  art 
as  science.  There  is  no  particular  method  or  design  that  that  is  necessarily  more 
appropriate  than  another.  The  choice  depends  on  a  variety  of  factors:  the  purpose  of  the 
study,  the  political  context,  the  money  and  time  available,  and  the  interests  and  abilities 
of  the  researcher  (Patton,  1990:  196).  No  method  is  inherently  superior  to  another,  but 
each  method  can  produce  a  different  twist  on  the  data.  The  challenge,  according  to 
Patton,  is  to  determine  what  information  is  needed  and  then  to  use  the  approach  that  is 
most  likely  to  produce  that  information  (Patton,  1990:  196). 

“Methodological  Mix”  and  Triangulation 

Patton  says  that  a  study  may  use  more  than  one  type  of  data  collection  strategy. 

In  fact,  he  stresses  that  the  reliability  and  validity  of  the  study  are  enhanced  through 
multiple  methods  of  data  collection  (Patton,  1990:  186-187).  To  strengthen  any  study 
design  and  to  more  adequately  address  the  question  of  causation,  Patton  recommends  the 
use  of  triangulation,  which  he  describes  as  a  “combination  of  methodologies  in  the  study 
of  the  same  phenomena  or  programs”  (Patton,  1990:  187).  Triangulation  can  be  achieved 
by  combining  different  kinds  of  qualitative  methods,  such  as  observation  and 
interviewing  or  interviewing  and  document  analysis,  or  by  “borrowing  and  combining 
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parts  from  pure  methodological  approaches,  thus  creating  mixed  methodological 
strategies”  (Patton,  1990:  188). 

What  are  the  “pure  methodological  approaches”  to  which  Patton  refers? 

Silverman  describes  four  broad  types:  observation,  interviewing,  textual  analysis,  and 
discourse  analysis  (Silverman,  1993:  9).  Yin  discusses  five  different  qualitative 
strategies:  experiment,  survey,  archival  analysis,  history,  and  case  study  (Yin,  1989:  17). 
In  Designing  Qualitative  Research.  Marshall  and  Rossman  list  14  different  techniques  for 
data  collection,  such  as  observation,  interview,  questiormaire,  etc.,  stating  that  each 
approach  is  best  suited  to  yield  a  particular  type  of  information.  The  researcher  must  pick 
the  “most  practical,  efficient,  and  feasible”  method  or  methods  (Marshall  and  Rossman, 
1989:  108). 

Sample  Size 

One  of  the  major  differences  between  quantitative  and  qualitative  research  is  the 
process  of  determining  sample  size.  Quantitative  studies  are  typically  based  on  relatively 
large  samples  that  are  selected  randomly.  Random  (or  probability)  sampling  is  vitally 
important  in  statistical  analysis  because  the  sample  results  will  be  used  to  infer  the 
characteristics  of  the  population  from  which  the  sample  was  drawn  (McClave  and 
Benson,  1994:  167).  The  purpose  of  probability  sampling,  then,  is  generalization  of 
results.  Specific  characteristics  of  the  sampled  data  are  not  of  primary  importance. 

The  purpose  of  qualitative  sampling,  on  the  other  hand,  is  to  select  “information- 
rich”  cases  for  in-depth  study  (Patton,  1990: 169).  Patton  refers  to  this  approach  as 
“purposeful  sampling” — ^the  selection  of  specific  cases  that  will  provide  insight  into  the 
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concepts  and  help  to  answer  the  questions  under  investigation  (Patton,  1990:  169).  To 
find  information-rich  cases,  Patton  suggests  simply  asking  the  right  questions  of  the  right 
people  (Patton,  1990:  176).  The  best  subjects  for  study  will  surface  as  more  and  more 
people  recommend  particular  cases,  programs,  people,  or  processes. 

The  question  arises.  How  big  should  the  sample  be?  Patton  answers  this  question 
forthrightly: 

There  are  no  rules  for  sample  size  in  qualitative  inquiry  (author’s  emphasis). 
Sample  size  depends  on  what  you  want  to  know,  the  purpose  of  the  inquiry, 
what’s  at  stake,  what  will  be  useful,  what  will  have  credibility,  and  what  can  be 
done  with  available  time  and  resources.  (Patton,  1990:  184) 

Sources  of  Evidence 

Yin  discusses  six  sources  of  evidence  for  collecting  qualitative  research  data, 

including  documentation  (Yin,  1989:  85).  Documentation  exists  in  many  forms,  such  as 

letters,  memorandums,  reports,  news  articles,  annoimcements,  etc.  The  primary  form  of 

documentation  relevant  to  this  research  is  what  Yin  calls  “administrative  documents” — 

proposals,  progress  reports,  and  other  internal  documents  (Yin,  1989:  85).  Yin  also 

discusses  the  use  of  interviews  as  a  source  of  evidence  (Yin,  1989:  85). 

Patton  talks  about  the  fact  that  program  files  and  documents  vary  greatly  in 

quality  and  may  be  incomplete  and  inaccurate  (Patton,  1989:  245).  Nevertheless, 

document  analysis  can  be  a  rich  source  of  information  to  augment  findings  from  other 

data  collection  methods  such  as  observation  or  interviews,  and  to  provide  a  direction  for 

further  research  (Patton,  1989:  245).  Patton  writes  the  following: 

Program  documents  provide  valuable  information  because  of  what  the  evaluator 
can  learn  directly  by  reading  them;  but  they  also  provide  stimulus  for  generating 
questions  that  can  only  be  pursued  through  direct  observation  and  interviewing. 
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Thus  program  records  and  documents  serve  a  dual  purpose:  (1)  they  are  a  basic 
source  of  information  about  program  decisions  and  background,  or  activities  and 
processes,  and  (2)  they  can  give  the  evaluator  ideas  about  important  questions  to 
pursue  through  more  direct  observations  and  interviewing.  (Patton,  1989:  233) 

Summary  of  Qualitative  Research 

In  summary,  qualitative  research  design  does  not  follow  a  specific  pattern  or 
approach.  The  design  will  depend  heavily  on  the  purpose  of  the  researcher,  the  subject 
under  study,  the  conditions  under  which  the  study  is  being  conducted,  and  the  time  and 
resources  available.  No  design  is  perfect,  as  Patton  points  out  (Patton,  1990:  162). 
Trade-offs  and  sacrifices  will  always  be  necessary  because  of  naturally  imposed  limits, 
the  greatest  of  these  limitations  perhaps  being  the  ability  of  the  human  mind  to  “grasp  the 
complex  nature  of  social  reality”  (Patton,  1990:  162). 

Selected  Approach 

It  was  determined  that  a  mixed  methodology  of  document  analysis  and  ihterviews 
was  the  best  approach  for  this  study.  The  combination  of  document  analysis  and 
interviews  was  necessary  to  help  identify  differences  in  approach  between  pre-FASA  and 
post-FASA  debriefings.  The  specific  approach  for  each  method  of  data  collection  is 
discussed  below. 

Document  Analysis 

Patton  says  it  is  very  important  at  the  beginning  of  a  research  project  to  negotiate 
access  to  pertinent  program  documents  and  records  and  to  respect  the  confidentiality  of 
the  information  (Patton,  1989:  233).  The  Staff  Summary  Sheet  (AF  Form  1768)  at 
Appendix  A  indicates  that  approval  was  obtained  to  review  and  copy  information  relevant 
to  this  study.  Because  a  large  body  of  documentation  was  collected  that  is  classified  as 
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government  “source  selection  sensitive,”  a  condition  for  access  to  the  data  was  that  all 
copies  must  be  destroyed  after  completion  of  the  research. 

The  primary  documents  selected  for  analysis  were  the  post-award  debriefing 
memorandums  prepared  by  the  contracting  officer.  The  research  plan  called  for 
comparison  of  pre-FASA  and  post-FASA  debriefing  memorandums,  using  the  FASA 
codification  date  of  1  October  1994  as  the  demarcation  between  pre-  and  post-FASA 
periods.  During  the  course  of  the  research,  however,  it  was  discovered  that  Air  Force 
policy  for  expanding  the  debriefing  process  actually  began  before  FASA  became 
effective.  A  policy  letter  issued  by  the  Assistant  Secretary  of  the  Air  Force  on  13 
October  1993,  based  largely  on  ASC  recommendations,  directed  major  changes  in  the 
conduct  of  post-award  debriefings  (see  Appendix  B).  The  changes,  though  not  exactly 
the  same,  were  very  similar  to  the  FASA  changes  that  became  law  one  year  later.  The 
major  difference  was  that  the  Air  Force  policy  did  not  specifically  address  whether 
release  of  overall  cost  and  technical  ratings  of  the  successful  offeror  was  permissible.  To 
eliminate  the  effect  of  the  Air  Force  policy  on  the  post-award  debriefing  documentation, 
no  debriefing  memorandums  between  the  period  of  October  1993  and  October  1994  were 
selected  for  analysis.  For  the  purposes  of  this  research,  the  pre-FASA  period  is  defined 
as  that  period  before  October  1993.  The  post-FASA  period  covers  the  time  after  October 
1994. 

The  post-award  debriefing  memorandums  were  collected  at  the  Aeronautical 
Systems  Center  Source  Selection  Support  Office  (ASC/SYG).  This  office  is  responsible 
for  managing  the  large-dollar  (over  $5  million),  “formal”  source  selections.  Because  of 
their  high  value,  only  a  relatively  small  number  of  these  source  selections  are  conducted 


32 


each  year.  When  the  source  selections  are  completed,  most  documentation  is  sent  to 
government  archives  for  long-term  storage.  Consequently,  the  data  directly  available, 
though  considered  acceptable  for  this  study,  was  limited  in  amount.  To  augment  the 
amount  of  data,  the  files  at  the  Operational  Contracting  Division  (ASC/PKW)  served  as  a 
valuable  back-up  resource.  Source  selections  at  ASC/PKW  are  smaller  and  less  formal, 
typically  ranging  from  $100,000  to  $5  million.  The  smaller  dollar  value  had  no  impact 
on  the  quality  of  the  information  in  the  post-award  debriefing  documentation. 

Personal/Telephone  Interviews 

According  to  Cooper  and  Emory,  personal  interviewing  is  an  excellent  method  of 
collecting  data  (Cooper  and  Emory,  1995:  270).  This  technique  can  provide  the 
researcher  with  information  of  great  depth  and  detail.  In  addition,  personal  interviews 
allow  the  researcher  to  take  steps  to  improve  the  quality  of  the  data  by  noting  specific 
conditions,  probing  with  additional  questions,  and  obtaining  more  information  through 
observation  (Cooper  and  Emory,  1995:  271).  Disadvantages  of  personal  interviews 
include  introduction  of  bias  through  inappropriate  facial  or  body  language,  word 
emphasis,  or  tone  of  voice  (Cooper  and  Emory,  1995:  278). 

Successful  personal  interviews  depend  on  three  broad  conditions: 

(1)  The  availability  of  the  required  information  from  the  respondent. 

(2)  The  respondent’s  imderstanding  of  his  or  her  role. 

(3)  Adequate  motivation  on  the  part  of  the  respondent. 

(Cooper  and  Emory,  1995: 271) 
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All  three  conditions  are  largely  under  the  control  of  the  researcher.  Screening  questions 
can  be  used  to  determine  the  respondent’s  knowledge  of  and  information  about  the  topic. 
The  researcher  should  also  take  time  to  explain  the  purpose  of  the  research,  the  kind  and 
extent  of  information  sought,  and  the  form  and  detail  of  the  answers.  Last,  the  researcher 
can  help  ensure  respondent  cooperation  by  being  friendly,  courteous,  and  confident 
(Cooper  and  Emory,  1995:  271-272). 

Time,  cost,  and  geographic  constraints  can  make  personal  interviewing  difficult 
(Cooper  and  Emory,  1995:  278).  Because  of  their  convenience  and  low  costs,  telephone 
interviews  have  become  a  popular  mode  of  data  collection  for  business  and  social 
research  (Cooper  and  Emory,  1995:  279). 

Telephone  interviews  obviously  do  not  permit  the  degree  of  rapport  that  can  be 
achieved  through  personal  interviews.  Research  has  shown  that  telephone  interviews 
result  in  less  detailed  responses,  and  they  are  generally  less  rewarding  and  enjoyable  to 
the  respondents  (Cooper  and  Emory,  1995:  281).  Also,  time  constraints  and  lack  of 
visual  aids  may  affect  the  quality  of  the  responses  (Cooper  and  Emory,  1995:  281). 

These  disadvantages  may  be  offset  by  the  fact  that  telephone  interviews  are  quicker  than 
personal  interviews,  and  they  may  reduce  interviewer  bias  (Cooper  and  Emory,  1995: 
280). 

Since  this  is  a  qualitative  study,  the  questions  were  not  designed  to  elicit 
responses  that  can  be  easily  quantified  and  tabulated.  All  questions  were  open-ended.  In 
other  words,  they  allowed  for  a  free  choice  of  words  rather  than  for  a  choice  among 
specified  alternatives  (Cooper  and  Emory,  1995:  311).  Respondents  answered  based  on 
their  subjective  opinions  and  evaluations  of  the  post-award  debriefing  process.  Silverman 
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refers  to  this  approach  as  interactionism,  the  attempt  to  generate  data  that  provide  an 
“authentic  insight  into  people’s  experiences”  (Silverman,  1995:  91). 

The  reliability  and  validity  of  open-ended  interviews  has  been  questioned  because 
of  the  lack  of  structure  and  standardization  (Silverman,  1995:  95).  To  increase  reliability 
and  validity,  the  interviews  were  of  the  type  Patton  calls  the  “standardized  open-ended 
interview”  (Patton,  1990:  289).  Patton  says  the  basic  purpose  of  the  standardized  open- 
ended  interview  is  to  “minimize  interviewer  effects  by  asking  the  same  question  of  each 
respondent”  (Patton,  1989:  285).  The  systematic  nature  of  the  standardized  open-ended 
interview  reduces  the  necessity  for  interviewer  judgment  and  also  helps  reduce  time  by 
focusing  the  parties  on  the  subject  at  hand  (Patton,  1989:  285). 

In  keeping  with  this  form  of  interview,  the  wording  and  sequence  of  the  questions 
was  determined  in  advance.  All  interview  participants  were  asked  the  same  questions  in 
the  same  order.  Questions,  as  stated  earlier,  were  completely  open-ended.  All  interviews 
were  limited  to  no  more  than  thirty  minutes  each.  Government  respondents  were  pre¬ 
screened  to  determine  their  knowledge  of  and  experience  with  the  post-award  debriefing 
process.  An  attempt  was  made  to  interview  government  personnel  who  had  conducted 
debriefings  both  before  and  after  FASA  implementation.  Likewise,  contractor  personnel 
having  both  pre-FASA  and  post-FASA  debriefing  experience  were  sought  to  the  extent 
possible  and  practical.  Interview  questions  are  listed  in  Appendix  C. 

Summary  of  Selected  Approach 

The  combination  of  document  analysis  and  personal/telephone  interview  was 
considered  to  be  the  best  approach  for  this  study,  given  the  time  and  fimding  constraints. 
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The  selected  methodology  was  designed  to  provide  a  framework  for  the  qualitative 
analysis  of  the  central  research  question:  What  has  been  the  effect  of  the  expanded  post¬ 
award  debriefing  process  on  the  likelihood  of  protest?  In  other  words.  Are  contractors 
less  likely  to  file  a  protest  under  the  new  debriefing  process  than  they  were  imder  the  old 
debriefing  process?  The  next  chapter  discusses  the  collection  of  data  to  answer  this  and 
other  research  questions. 
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IV.  Data  Collection  and  Analysis 


Introduction 

A  total  of  1 6  post-award  debriefing  documents  were  analyzed,  eight  dated  before 
FASA  and  eight  dated  after  FASA.  Most  of  the  acquisitions  were  fixed-price  supply,  but 
there  were  some  service  efforts.  The  dollar  values  of  the  actions  ranged  fi-om  $1  million 
to  over  $100  million.  Ten  companies  were  large  businesses  and  six  were  small 
businesses. 

To  augment  the  document  analysis,  13  interviews  were  conducted  during  the 
period  16  June  1997  through  20  June  1997.  Eight  government  and  five  industry 
representatives  participated.  Four  government  interviews  were  personal  interviews  and 
four  were  conducted  by  telephone.  All  contractor  interviews  were  telephone  interviews. 
Both  government  and  industry  participants  were  asked  the  same  questions,  with  only  the 
necessary  changes  being  made.  Government  and  industry  participants  are  listed  in 
Appendix  D. 

All  participants  were  apprised  of  the  academic  nature  of  this  project  and  were 
assured  of  confidentiality  and  non-attribution.  Most  participants  were  aware  of  the  Air 
Force  policy  on  post-award  debriefings  that  was  issued  in  October  1993.  Not  all  could 
recall,  however,  whether  they  had  participated  in  debriefings  during  the  period  from 
October  1 993  to  October  1 994,  when  FASA  became  effective.  Consequently,  it  was  not 
possible  to  eliminate  potential  effects  of  the  Air  Force  policy  during  that  one-year  period. 
The  interview  question  most  likely  to  be  affected  was  question  3. 
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Results  of  Document  Analysis 


Post-award  debriefing  documents  were  analyzed  by  comparing,  to  the  extent 
possible,  the  process  and  content  of  both  the  old,  pre-FASA  debriefings  and  the  new, 
post-FASA  debriefings.  The  results  of  the  analysis  indicated  three  major  areas  of 
difference  between  the  old  and  new  processes: 

(1)  Comparison  of  overall  cost  and  technical  ratings  of  debriefed  offeror 
and  successful  offeror. 

(2)  Summary  of  the  rationale  for  the  award. 

(3)  Opportunity  for  questions  and  answers. 

Each  area  of  difference  is  discussed  in  the  following  sections. 

Comparison  of  Cost  and  Technical  Ratings 

It  is  evident  from  an  examination  of  the  debriefing  documents  that  the  old  process 

did  not  compare  the  overall  cost  and  technical  ratings  of  the  debriefed  offeror  and  the 
successful  offeror.  This  fact  does  not  come  as  a  surprise  because  the  FAR  at  that  time  did 
not  require  the  contracting  officer  to  address  the  winning  offer  in  any  fashion.  In  fact, 
comparison  of  offers  was  strictly  forbidden.  FAR  15.1003(b)  contained  the  following 
statement:  “Debriefing  shall  not  reveal  the  relative  merits  or  technical  standing  of 
competitors  or  the  evaluation  scoring.” 

On  the  other  hand,  the  FAR  now  encourages  a  comparison  of  the  overall  cost  and 
technical  ratings  of  the  winning  proposal  and  the  debriefed  offeror’s  proposal.  FAR 
15.1006(d)(2)  states  that,  at  a  minimum,  the  debriefing  shall  include  the  “...overall  cost 
or  price  and  technical  rating,  if  applicable,  of  the  successful  offeror  and  the  debriefed 
offeror. . . .”  Post-award  debriefing  documents  under  the  new  process  indicated  a  gradual 
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approach  toward  providing  information  on  the  winning  proposal.  Of  the  eight  post- 
FASA  debriefing  records  examined,  two  did  not  include  information  on  the  successful 
offeror.  These  two  were  dated  within  the  first  six  months  after  FASA  implementation. 
The  six  remaining  records  dated  more  than  six  months  after  FASA,  however,  clearly 
provided  the  cost  and  technical  ratings  of  the  winner.  Apparently,  it  took  some  time  for 
all  government  personnel  to  learn  what  was  acceptable  in  the  new  process  and  to 
overcome  the  inertia  of  the  old  way.  It  is  interesting  to  note  that  one  of  the  debriefed 
offerors  who  did  not  receive  information  on  the  wiiming  proposal  requested  the 
information  during  the  question-and-answer  period.  The  offeror  weis  told  that  release 
would  have  to  be  cleared  “through  legal.”  No  record  was  found  indicating  whether  the 
information  was  subsequently  provided. 

Summary  of  Rationale  for  Award 

A  similar  pattern  is  evident  concerning  release  of  the  summary  of  the  rationale  for 
award.  No  pre-FASA  records  included  award  rationale  because  this  information  was  not 
allowed  by  FAR  before  the  implementation  of  FASA.  Of  the  eight  post-FASA  records, 
three  included  the  source  selection  decision  memorandum  and  two  included  a  summary 
of  the  basis  for  the  award.  One  record  indicated  that,  as  a  result  of  a  request  by  the 
debriefed  offeror,  the  information  would  be  provided.  The  remaining  two  records  under 
the  new  process  were  silent  on  award  rationale.  These  documents  were  dated  within  the 
first  five  months  of  FASA. 
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Opportunity  for  Questions  and  Answers 


Perhaps  the  most  notable  difference  between  the  old  and  new  processes  was  in  the 
willingness  of  government  personnel  to  entertain  questions  from  the  debriefed  offerors. 
Even  though  FAR  did  not  specifically  prohibit  questions  under  the  old  process,  none  of 
the  debriefing  files  from  the  period  before  FASA  contained  any  record  of  a  question-and- 
answer  period.  Neither  did  any  of  the  debriefing  scripts  from  this  period  indicate  a 
willingness  on  the  part  of  government  personnel  to  address  questions.  By  contrast,  all 
eight  records  in  the  post-FASA  period  indicated  in  some  fashion,  either  through  scripts, 
briefing  charts,  or  actual  question-and-answer  lists,  that  questions  were  permitted  and,  in 
some  cases,  actively  encouraged. 

As  mentioned  above,  FAR  did  not  specifically  prohibit  questions  in  post-award 
debriefings  prior  to  FASA,  but  neither  did  it  specifically  allow  them.  Findings  in  the 
literature  review  suggested  that  debriefings  before  FASA  were  largely  pro  forma,  with 
little  or  no  useful  information  provided.  Questions  were  no  doubt  asked  during 
debriefings  and,  as  long  as  they  pertained  to  the  debriefed  offeror’s  proposal,  they  were 
probably  answered.  The  climate  surrounding  post-award  debriefings  in  that  period, 
however,  did  not  encourage  risk  taking  by  government  personnel.  As  a  consequence,  no 
contracting  officer  before  FASA  was  likely  to  encourage  questions  of  any  kind  during  a 
debriefing,  much  less  actually  record  them.  The  new  process,  on  the  other  hand, 
encourages  questions.  FAR  15.1006(d)(6)  now  allows  “(r)easonable  responses  to 
relevant  questions  about  whether  source  selection  procedures  contained  in  the 
solicitation,  applicable  regulations,  and  other  applicable  authorities  were  followed.”  The 
phrase  “reasonable  responses  to  relevant  questions”  is  susceptible  to  a  wide  range  of 
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interpretation,  but  the  post-FASA  climate  would  seem  to  favor  more  openness  and  less 
risk  aversion. 

Summary  of  Document  Analysis 

Document  analysis  provided  a  good  basis  for  imderstanding  the  debriefing  process 
in  general,  and  in  assessing  the  differences  between  the  old  and  new  debriefing  processes. 
Major  differences  in  three  areas  were  evident:  (1)  a  comparison  of  the  imsuccessflil 
offeror’s  cost  and  technical  ratings  with  those  of  the  successful  offeror;  (2)  a  summary  of 
the  rationale  for  award;  and  (3)  an  opportunity  for  questions  and  answers. 

The  analysis  was  not  sufficient,  however,  to  answer  the  primary  questions  of  this 
study.  There  was  no  information  in  the  documentation  to  determine,  for  example,  if 
unsuccessful  offerors  are  less  likely  to  protest  because  of  the  new  debriefing  process. 
Also,  it  could  not  be  determined  in  the  post-award  records  whether  the  new  process 
enables  unsuccessful  offerors  to  develop  more  competitive  proposals  for  future 
procurements.  In  addition,  document  analysis  alone  did  not  provide  enough  information 
to  recommend  improvements  in  the  current  debriefing  process.  The  answers  to  these 
primary  questions  depended  on  the  interview  findings. 

Interview  Results 

Question  1 :  The  Federal  Acquisition  Streamlining  Act  of  1994  (FAS A)  directed 
many  changes  to  the  government  acquisition  process,  among  them  the  expansion 
of  the  post-award  debriefing  process. 

a.  Have  you  conducted  or  participated  in  a  post-award  debriefing  of  an 
unsuccessful  offeror? 

All  participants,  both  government  and  industry,  answered  “yes”  to  this  question. 
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b.  Has  your  participation  included  experience  both  before  and  after  FASA? 

All  participants  answered  “yes.” 

Question  2:  FASA  requires  that  post-award  debriefings  address  significant  weak 
or  deficient  factors  in  the  unsuccessful  offeror’s  proposal. 

a.  In  your  opinion,  do  the  current  debriefings  clearly  identify  proposal 
weaknesses  and  deficiencies? 

Most  government  participants  answered  “yes”  to  this  question.  Some  gave 
qualified  responses,  saying  that  the  government  identifies  proposal  weaknesses  and 
deficiencies  to  the  extent  it  can  without  resorting  to  technical  leveling.  Others  said  the 
answer  depends  on  who  does  the  debriefing.  In  too  many  cases,  the  old  perspective  still 
prevails — the  government  is  reluctant  to  be  frank  and  forthcoming.  One  respondent  said, 
“Changing  the  law  has  not  changed  the  mindset.”  Another  said  there  is  too  much 
“gaming”  or  rigging  of  the  technical  evaluation  process  to  make  favored  companies  look 
good  and  less  favored  companies  look  bad.  In  these  situations,  the  contracting  officer  has 
a  difficult  time  clearly  identifying  and  explaining  proposal  weaknesses  and  deficiencies 
to  an  unsuccessful  offeror. 

Most  industry  participants  also  answered  “yes.”  One  said  he  is  reasonably 
satisfied  that  proposal  weaknesses  and  deficiencies  are  addressed,  but  believes  the 
process  “is  all  subjective  anyway.”  Another  answered  clearly  “no.”  He  said  acquisition 
reform  has  improved  the  process,  but  there  is  still  so  much  fear  of  protest  that  government 
personnel  adopt  a  too  conservative  approach.  They  do  not  address  proposal  weaknesses 
and  deficiencies  to  the  extent  the  law  now  allows. 
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b.  Comparing  the  process  before  and  after  FASA,  do  you  see  a  difference  in  the 
way  debriefings  have  been  conducted  in  this  area? 

All  government  participants  except  one  answered  “yes”  to  this  question.  The 

consensus  was  that  the  government  is  now  much  more  likely  to  elaborate  on  the 

weaknesses  and  deficiencies  and  to  provide  detailed  information  on  the  ratings.  Before 

FASA,  only  a  minimal  amount  of  information  was  provided.  Often,  in  pre-FASA 

debriefings,  the  only  explanation  was  that  “you  just  weren’t  good  enough  in  this  area.” 

The  lone  government  dissenter  on  this  question  said  that  the  government  has  not  changed 

its  approach — it  is  still  clinging  to  its  pre-FASA  ways. 

Most  industry  participants  said  the  government  has  made  a  noticeable 

improvement  in  identifying  and  explaining  weaknesses  and  deficiencies.  The 

government  is  much  more  open  and  forthcoming  in  divulging  award  criteria  and  in 

explaining  where  the  unsuccessful  offeror  “missed  the  mark”  in  comparison  to  the 

winning  proposal.  One  respondent  said  that,  while  the  government  has  made  positive 

changes  in  this  area,  the  color  ratings  are  too  broad  to  allow  intelligent  distinctions. 

Question  3:  FASA  requires  disclosure  of  the  overall  evaluated  cost  and  technical 
ratings  of  the  successful  offeror  and  the  tmsuccessful  offeror. 

a.  In  your  opinion,  do  debriefings  clearly  show  the  overall  cost  and  technical 
ratings  of  the  successful  offeror? 

Government  participants  generally  believed  that  most  debriefings  now  show  the 
overall  cost  and  technical  ratings  of  the  successful  and  unsuccessful  offerors.  Some 
questioned  the  value  of  the  color  codes,  however.  They  said  the  codes  are  too  “top 
level” — i.e.,  color  coding  does  not  really  convey  much  information.  When  both  winner 
and  loser  are  coded  the  same  in  a  particular  area,  it  is  very  difficult  to  give  the  loser  a 
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good  explanation  of  why  he  lost.  Too  much  detail  runs  the  risk  of  technical  leveling  or 
of  divulging  proprietary  information. 

Most  industry  participants  answered  “yes”  or  “qualified  yes.”  Two  said  the 
overall  ratings  are  too  broad  to  be  helpful — that  more  detail  is  needed.  One  suggested  a 
price  breakout  by  line  item.  Another  said  it  is  fhistrating  to  study  the  RFP,  interpret  what 
the  government  wants,  build  the  proposal  based  on  customer  need,  and  then  lose  to 
someone  who  has  interpreted  the  requirement  another  way.  He  said  this  situation  leaves 
him  wondering  if  he  bid  properly,  if  he  just  made  a  mistake,  or  if  the  successful  offeror 
“bent  the  rules”  in  some  fashion.  A  detailed  explanation  of  the  cost  and  technical  ratings 
would  be  helpful  in  these  cases. 

b.  Comparing  the  process  before  and  after  FASA,  do  you  see  a  difference  in  the 

way  debriefings  have  been  conducted  in  this  area? 

Government  participants  said  there  is  a  big  difference.  Before  FASA,  the 
successful  offeror’s  cost  and  technical  ratings  were  not  provided  and  the  winning 
proposal  was  not  discussed  at  all.  Now,  the  government  provides  the  successful  offeror’s 
ratings  and  is  willing  to  discuss  the  merits  of  the  winning  proposal  within  the 
“appropriate  bounds.”  The  post-FASA  government  approach  is  much  more  helpful  and 
“user  friendly”  in  discussing  the  successful  offeror’s  cost  and  technical  ratings.  One 
government  respondent  said  there  is  now  a  sincere  desire  on  the  part  of  the  government  to 
be  as  open  and  frank  as  possible  in  order  to  help  unsuccessful  offerors  prepare  better 
proposals  in  future  procurements. 

Industry  respondents  were  clearly  divided  in  this  area.  Some  saw  a  big  change 
from  the  old  process.  One  commented  that  the  new  process  provides  a  sense  of  relative 
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standing — ^there  is  something  to  compare  his  proposal  against.  Others  said  there  is  not 
much  difference  from  the  old  process.  Technical  issues  may  be  more  open,  but  cost 
issues  are  still  not  truly  discussed.  One  respondent  said  he  has  quit  going  to  debriefings 
because  they  are  “worthless.”  The  color  codes  of  the  successful  offeror  provide  little 
useful  information,  and  the  selection  decision  is  too  often  a  “political  call.” 

Question  4:  FAS  A  requires  disclosure  of  the  overall  ranking  of  all  offerors  £ind  a 

summary  of  the  rationale  for  award. 

a.  In  your  opinion,  do  debriefings  clearly  provide  the  overall  rankings  of  the 
offers  and  a  summary  of  the  rationale  for  award? 

Many  government  participants  pointed  out  that  the  specific  FASA  requirement  is 
to  show  the  overall  ranking  of  all  offerors  when  any  ranking  was  developed  by  the 
agency.  It  is  Air  Force  policy  not  to  rank  offerors,  therefore  no  overall  ranking  is 
required.  One  respondent  said  that  there  are  only  two  categories  of  offerors  in  an  Air 
Force  debriefing:  “winner  and  loser.”  Some  said  the  number  of  offerors  is  divulged  but 
not  the  names.  Others  said  that  all  unsuccessful  offerors  are  listed  in  alphabetical  order, 
but  no  information  about  them  is  provided  to  the  offeror  being  debriefed.  One 
maintained  that  rankings  should  be  used  because  it  is  important  that  the  debriefed  offeror 
know  his  relative  position  among  all  the  offerors. 

Nearly  all  government  participants  indicated  that  some  form  of  award  rationale  is 
provided  to  the  debriefed  offeror.  Most  said  the  source  selection  decision  document  is 
almost  always  used,  but  sometimes  a  redacted  summary  is  provided.  One  said  award 
rationale  has  always  been  provided,  but  under  the  new  rules  more  detail  is  necessary. 
Another  commented  that  the  quality  and  usefulness  of  the  source  selection  decision 
document  has  suffered  because  of  the  possibility  of  release  of  the  information. 
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Some  industry  participants  said  they  have  seen  rankings  in  Air  Force  debriefings, 
but  others  said  the  offerors  were  listed  but  not  ranked.  Nearly  all  said  rankings  would  be 
helpful  in  determining  how  close  they  were  to  winning.  Knowing  their  relative  position 
would  help  them  decide  whether  they  are  competitive  enough  to  pursue  other,  similar 
contracts. 

All  industry  respondents  said  that  the  government  should  reveal  as  much  as 
possible  about  the  reasoning  behind  the  award  decision.  Two  expressed  clear 
dissatisfaction  with  the  quality  of  the  award  rationale  they  had  received.  One  said 
government  personnel  are  reluctant  to  disclose  source  selection  decision  documents 
because  of  a  fear  of  protest,  and  that  redacted  summaries  normally  do  not  provide  enough 
information  to  be  of  any  value.  Another  respondent  said  that,  on  three  different 
occasions,  he  was  rated  technically  acceptable  and  had  a  lower  price  than  the  successful 
offeror,  but  lost  all  three  awards.  He  said  he  was  never  given  a  good  explanation  of  why 
he  lost,  so  he  just  assumed  the  government  wanted  someone  else. 

b.  Comparing  the  process  before  and  after  FASA,  do  you  see  a  difference  in  the 

way  debriefings  have  been  conducted  in  this  area? 

Most  government  participants  saw  a  major  difference,  especially  in  the  release  of 
award  rationale.  One  commented  that,  under  the  old  process,  the  source  selection 
decision  document  was  treated  as  a  highly  sensitive,  almost  classified  document.  Now 
we  freely  distribute  it  at  post-award  debriefings.  Another  said  the  government  is  much 
more  willing  today  to  discuss  the  whole  source  selection  process. 

Industry  participants  generally  thought  the  new  process  is  better.  One  commented 
that  there  has  been  a  “progressive  change  for  the  better,”  especially  in  providing  award 
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rationale.  In  the  current  environment,  debriefings  provide  “pertinent  information”  on  the 
selection  decision.  Another  said  there  is  little,  if  any,  difference  in  this  area.  Contracting 
officers  still  defer  to  their  legal  counsel — ^there  is  too  much  “We’ll  have  to  look  into  that 
and  get  back  to  you.” 

Question  5:  FAS  A  requires  reasonable  responses  to  relevant  questions  concerning 
the  conformity  of  selection  procedures  to  the  solicitation,  regulations,  and  other 
applicable  authorities. 

a.  In  your  opinion,  do  Air  Force  personnel  allow  relevant  questions  and  provide 
reasonable  answers  during  the  post-award  debriefing? 

Generally,  government  personnel  said  they  encourage  questions  and  try  as  best 
they  can  to  answer  them.  Overall,  they  expressed  concern  that  the  words  “relevant”  and 
“reasonable”  are  subject  to  a  wide  range  of  interpretation.  One  said  generic  questions 
about  the  RFP  source  selection  procedures,  evaluation  criteria,  etc.,  are  allowed  and 
addressed,  but  specific  questions  on  the  successful  offeror’s  proposal  and  other  offerors’ 
proposals  are  not  allowed.  Another  commented  that  contractors  are  using  “reasonable 
responses  to  relevant  questions”  as  a  means  to  “see  how  far  they  can  push  the  process  to 
get  what  they  need.” 

For  the  most  part,  industry  participants  agreed  that  government  persormel  try  to 
respond  reasonably  to  relevant  questions.  Again,  the  interpretation  of  “reasonable”  and 
“relevant”  was  an  issue.  One  industry  representative  said  questions  are  allowed  but 
reasonable  answers  are  not  always  provided  because  of  fear  of  “giving  too  much 
information  or  the  wrong  information.”  Another  said  some  relevant  questions  are  not 
answered,  but  reasonable  answers  are  given  for  those  questions  the  government  “chooses 
to  answer.”  Another  responded  that  the  government  is  probably  as  “candid  and  open  as  it 
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can  be.”  There  are  specific  areas,  such  as  “cost  issues  and  proprietary  data,”  that  are 
“difficult  to  address.” 

b.  Comparing  the  process  before  and  after  FASA,  do  you  see  a  difference  in  the 

way  debriefings  have  been  conducted  in  this  area? 

Government  personnel  said  this  area  represents  a  big  change  from  the  old 
process.  In  the  pre-FASA  era,  there  was  a  “strained  atmosphere”  and  a  “set  agenda.” 

The  debriefed  offeror  could  ask  questions  about  his  proposal  only.  Now,  the  process  is 
more  open  and  relaxed.  Questions  about  the  successful  offeror’s  proposal  can  be  asked, 
as  well  as  questions  about  the  selection  process  in  general.  Most  government  participants 
said  they  encourage  questions  under  the  new  rules.  Some  expressed  concern  that  they 
had  gone  too  far  in  answering  imsuccessful  offerors’  questions.  One  said  some 
“discipline”  needs  to  be  added  to  the  process  “to  keep  things  from  getting  out  of  line” 
where  questions  and  answers  are  concerned.  According  to  one  participant,  the 
contractors’  approach  now  is  “nothing  ventured,  nothing  gained” — ^an  apparent  reference 
to  the  perceived  boldness  of  industry  questions. 

Industry  responses  to  this  question  were  mixed.  Two  participants  said  there  is 
now  a  genuine  desire  and  attempt  by  the  government  to  address  relevant  questions. 
Another  said  there  is  little  difference — ^the  government’s  freedom  to  answer  questions  in 
debriefings  has  always  been  “bound  by  laws  and  regulations.”  One  said  there  is  no 
difference.  The  intent  to  answer  forthrightly  might  be  present,  but  the  actual  answer  is 
too  often  “circumscribed  by  fear.” 
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Question  6:  What,  in  your  opinion,  is  the  biggest  difference  between  the  old 
debriefing  process  and  the  new  debriefing  process? 

The  general  consensus  of  government  persoimel  was  that  the  new  process  is  much 
more  open  and  relaxed.  One  said  the  whole  atmosphere  is  now  “less  adversarial.” 
Another  said  debriefings  before  FASA  were  “short  but  not  sweet.”  There  was  nothing 
much  to  discuss  beyond  the  unsuccessful  offeror’s  proposal,  and  even  then  the  discussion 
was  not  always  open  and  honest.  Government  personnel  dreaded  the  debriefings,  and  the 
contractors  often  left  feeling  “dissatisfied  and  disgruntled.”  Now,  the  government  often 
travels  to  the  unsuccessful  offeror’s  facility  for  the  debriefing,  as  an  “olive  branch”  to  the 
loser.  Most  government  participants  said  they  put  much  more  effort  into  debriefings  now 
than  before.  Debriefings  are  considered  to  be  a  much  more  important  part  of  the  job. 

Most  industry  participants  agreed  that  the  major  difference  is  openness  on  the  part 
of  the  government  to  discuss  more  areas  of  the  source  selection,  especially  the  successful 
offeror’s  proposal.  One  respondent  said  he  attended  debriefings  in  the  past  “just  in  case” 
there  was  some  useful  information.  Now  he  attends  because  he  knows  that  substantive 
issues  beyond  his  proposal  will  be  addressed.  Debriefings  are  much  more  useful  and 
worthwhile  today. 

Question  7:  Overall,  are  you  satisfied  that  the  current  post-award  debriefing 
process  provides  as  much  information  as  possible,  given  that  government 
personnel  must  conform  to  applicable  authorities  and  not  divulge  confidential  or 
proprietary  information? 

Five  government  participants  indicated  satisfaction  with  the  current  process  of 
debriefing.  Typical  comments  were  that  debriefings  provide  the  “most  information 
possible”  or  the  “right  amount”  of  information.  One  said  perhaps  too  much  information 
is  now  being  released,  especially  cost  information.  Another  expressed  concern  that  more 


49 


information  might  have  a  “deleterious  effect”  on  full  and  open  competition.  In  other 
words,  companies  might  fear  the  release  of  proprietary  data,  trade  secrets,  or  other 
sensitive  information,  and  become  reluctant  to  bid. 

Most  industry  participants  indicated  a  strong  preference  for  the  new  process  over 
the  old  process.  They  generally  agreed  that  the  new  process  provides  as  much 
information  as  possible  under  the  circumstances.  One  said  the  process  must  be 
informative,  but  the  govermnent  needs  to  be  careful  not  to  compromise  the  “competitive 
advantage”,  i.e.,  confidential  and  proprietary  information,  trade  secrets,  etc.  Another 
respondent  said  he  is  not  satisfied — government  personnel  are  still  overly  conservative. 
“Fear  of  retribution”  on  the  part  of  government  personnel  “defeats  the  purpose  of  FASA.” 

Question  8:  Do  you  believe  the  current  process  provides  information  that  leads  to 

the  development  of  more  responsive  and  competitive  proposals? 

Government  personnel  were  divided  on  this  question.  Three  said  the  current 
process  leads  to  better  proposals,  especially  when  there  are  recurring  source  selections  for 
similar  efforts.  Two  others  said  the  process  does  not  necessarily  produce  better  proposals 
because  most  debriefing  issues  and  questions  are  focused  on  the  source  selection  process 
itself,  and  not  on  how  to  improve  the  xmsuccessful  offeror’s  proposal.  One  response  was 
that  industry  proposal  writers  already  do  such  an  excellent  job  that  there  is  “not  much 
learning  ciuve  in  this  process.”  Another  participant  said  the  process  is  too  flawed  to 
produce  better  proposals.  Government  personnel  are  “not  honest  enough”  in  explaining 
proposal  deficiencies  to  allow  improvement. 

Three  industry  participants  answered  “yes”  to  this  question.  More  information 
leads  to  better  proposals.  Two  had  qualified  responses.  One  said  the  government  is 
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moving  in  the  right  direction,  but  needs  to  provide  more  information  on  the  successful 

offeror.  He  said  the  government  is  probably  reluctant  to  reveal  too  much  information  for 

fear  of  “exposing  a  possible  mismatch  between  the  RFP  requirements  and  the  evaluation 

of  the  successful  offeror.”  Another  respondent  said  the  real  key  to  improving  proposals 

is  to  get  as  much  information  on  the  requirement  as  early  as  possible  in  the  process. 

Question  9:  Compared  to  the  old  process  before  FAS  A,  do  you  believe 
information  provided  in  the  current  post-award  debriefing  process  has  reduced  the 
likelihood  of  protest  by  an  unsuccessful  offeror? 

Again,  government  personnel  were  divided  on  this  question.  Four  said  the 
likelihood  of  protest  has  definitely  been  reduced  because  companies  now  have  a  better 
understanding  of  the  rationale  for  award — ^they  do  not  need  to  protest  simply  to  gain  more 
information.  One  respondent  said  the  new  debriefings  have  had  “no  impact”  on  protests. 
Two  other  respondents  expressed  the  belief  that  protests  have  not  been  reduced  because 
more  information  only  “fuels  the  fire”  of  a  company  bent  on  protesting.  Another  said  the 
likelihood  of  protest  has  probably  increased  because  of  the  openness  of  the  debriefings. 
Now  that  the  government  has  “opened  up  the  gates,”  allowing  questions  on  “process  and 
procedure,”  contractors  are  taking  advantage  of  the  situation.  They  are  using  the 
additional  information  against  us  in  their  protests. 

Industry  responses  were  generally  positive — a  better  imderstanding  of  the  award 
decision  leads  to  a  reduced  likelihood  of  protest.  One  respondent  said  that,  “absent 
procedural  problems,”  an  unsuccessful  offeror  will  not  protest  if  he  has  a  clear 
understanding  of  why  he  lost.  For  this  reason,  it  is  important  for  government  personnel  to 
put  themselves  “in  the  shoes  of  the  loser”  when  preparing  a  debriefing.  Another  industry 
participant  said  the  new  process  forces  the  government  personnel  to  do  a  better  job  of 


51 


evaluating  proposals,  because  mistakes  will  eventually  “come  to  light.”  As  the  new 
process  becomes  more  ingrained,  protests  should  “go  down  markedly.”  In  the  words  of 
another  respondent,  the  new  process  is  a  “two-edged  sword.”  The  candid  nature  of  the 
debriefing  has  probably  kept  some  from  protesting,  but  may  have  “stoked  the  protest 
fires”  of  others. 

Question  10:  Do  you  have  any  specific  recommendations  as  to  how  the  current 

debriefing  process  can  be  improved? 

Most  government  participants  said  the  current  process  is  acceptable.  Some 
cautioned  against  making  it  “too  objective.”  The  strength  of  the  process  lies  in  its 
subjectivity,  i.e.,  in  the  ability  of  the  source  selection  authority  to  make  value  judgments. 
One  respondent  said  that  injecting  “more  honesty  into  the  proposal  evaluation  process” 
will  improve  debriefings  by  allowing  a  more  realistic  assessment  of  unsuccessful 
offerors’  proposals.  Another  recommendation  was  to  avoid  going  too  far  in  providing 
information  on  the  successful  offeror  and  risking  a  damaging  reaction  to  the  process.  Yet 
another  recommendation  was  to  provide  contractors  with  some  type  of  training  or 
workshop  on  how  the  government  does  source  selections.  Perhaps  the  training  could  be 
incorporated  into  an  AFMC  “road  show”  for  industry. 

Industry  participants  stated  a  general  preference  for  including  relative  rankings  of 
all  offerors.  Rankings  give  companies  a  better  idea  of  how  they  fared  against  their  peers 
and  whether  they  should  pursue  contracts  that  are  similar  in  nature.  Some  said  the 
government  should  hold  “industry  day”  workshops  that  focus  on  the  particulars  of 
proposal  evaluation,  especially  in  the  area  of  program  management.  Another  said  the 
government  should  engage  in  “role  playing”  when  preparing  for  a  debriefing. 
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Government  personnel  would  act  in  the  role  of  a  losing  contractor  in  a  post-award 
debriefing.  Role  playing  would  help  the  government  determine  if  the  information  in  the 
debriefing  will  answer  an  unsuccessful  offeror’s  questions.  A  final  recommendation  was 
for  the  government  to  “examine  FASA  more  closely.”  A  closer  look  at  FASA  would 
show  the  government  that  it  must  continue  to  move  away  fi’om  playing  “close  to  the  vest” 
and  toward  releasing  more  information.  Now  is  not  the  time  for  the  government  to  “rest 
on  its  laurels”  and  stop  improving  the  process. 

Summary 

This  chapter  described  the  data  collection  and  analysis  procedures  used  in  the 
course  of  this  research  project.  The  primary  methods  of  data  collection  were  document 
analysis  and  interviews.  Results  of  the  data  analysis  were  presented. 

Document  analysis  pinpointed  three  major  areas  of  difference  between  the 
old  and  new  debriefing  processes:  1)  comparison  of  overall  cost  and  technical  ratings  of 
debriefed  offeror  and  successful  offeror;  2)  summary  of  the  rationale  for  the  award,  and 
3)  opportunity  for  questions  and  answers.  The  new  debriefing  process  was  clearly 
superior  in  all  three  areas. 

Interview  results  indicated  a  wide  range  of  beliefs  and  perceptions  concerning  the 
differences  between  the  old  and  new  processes  and  the  effectiveness  of  the  new  process. 
Most  interview  participants  considered  the  current  practice  of  debriefings  to  be  better 
than  the  old  one  in  the  majority  of  the  areas  that  were  compared.  There  was  a  wide  range 
of  opinion  in  many  areas,  however. 
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Overall,  the  new  debriefing  process  appears  to  be  much  less  formal  and  rigid  and 
much  more  dynamic  and  wide-ranging.  The  next  chapter  will  present  specific 
conclusions  and  recommendations  from  the  findings,  along  with  suggested  areas  for 


further  research. 


V.  Conclusions  and  Recommendations 


Introduction 

As  Patton  observes,  the  purpose  of  qualitative  inquiry  is  to  produce  findings 

(Patton,  1990:  371).  Data  collection  is  not  an  end  in  itself.  It  culminates  in  analyzing  and 

interpreting  the  data,  presenting  the  findings,  and  drawing  conclusions  (Patton,  1990: 

371).  The  previous  chapter  was  concerned  with  data  collection  and  analysis.  The  focus 

of  this  chapter  is  on  conclusions  from  the  data  and  making  recommendations. 

Patton  also  observes  that  there  are  no  formulas  for  determining  the  significance  of 

the  data,  no  “agreed-on  canons  for  qualitative  analysis,  in  the  sense  of  shared  grovmd 

rules  for  drawing  conclusions  and  verifying  their  sturdiness”  (Patton,  1990:  372).  The 

goal  of  data  analysis,  according  to  Patton,  is  to  “fairly  represent  the  data  and 

communicate  what  the  data  reveal  given  the  purpose  of  the  study”  (Patton,  1990;  372). 

A  final  quote  from  Patton  would  appear  to  be  relevant  to  this  research  project: 

It  is  important  to  understand  that  the  interpretive  explanation  of  qualitative 
analysis  does  not  yield  knowledge  in  the  same  sense  as  a  quantitative  explanation. 
The  emphasis  is  on  illumination,  understanding,  and  extrapolation  rather  than 
causal  determination,  prediction,  and  generalization.  (Patton,  1990:  424) 

Review  of  Research  Objectives  and  Purpose  of  the  Study 

As  discussed  in  the  Literature  Review,  bid  protests  are  expensive,  burdensome, 
and  time-consuming  for  both  government  and  industry.  One  of  the  major  reasons  behind 
acquisition  reform  was  to  find  ways  to  reduce  the  number  of  protests  in  government 
acquisition.  A  suggested  approach  was  to  expand  the  post-award  debriefing  process  to 
give  unsuccessful  offerors  more  information  and  insight  into  the  government  source 
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selection  process.  A  primary  objective  of  this  research  was  to  determine  whether  FASA 
changes  in  the  post-award  debriefing  process  have  resulted  in  fewer  bid  protests  by 
unsuccessful  offerors.  Another  major  objective  was  to  determine  if  the  new  debriefing 
process  has  motivated  contractors  to  produce  better  proposals.  Better  proposals  help 
streamline  the  acquisition  process  by  reducing  government  evaluation  time.  They  also 
strengthen  the  industrial  base  by  stimulating  competition  for  government  contracts. 

Another  purpose  of  this  study  was  to  examine  the  current  post-award  debriefing 
process,  noting  differences  from  the  old  process  as  indicated  by  government  and  industry 
personnel  having  experience  with  debriefings  before  and  after  FASA.  A  final  purpose 
was  to  make  recommendations,  based  on  the  findings,  to  improve  the  debriefing  process. 

As  discussed  above,  this  chapter  will  present  the  research  conclusions  based  on 
findings  from  the  document  analysis  and  the  results  of  the  interviews.  Conclusions  will 
be  in  the  form  of  answers  to  the  investigative  questions  listed  in  Chapter  I.  In  addition, 
this  chapter  will  provide  recommendations  for  improving  the  post-award  debriefing 
process  and  also  suggest  related  areas  for  future  research. 

Investigative  Question  1 

How  does  the  new  post-award  debriefing  process  differ  from  the  old,  pre-FASA 

process? 

Document  analysis  indicated  three  major  areas  of  difference:  (1)  comparison  of 
overall  cost  and  technical  ratings  of  the  debriefed  offeror  and  the  successful  offeror; 

(2)  summary  of  the  rationale  for  the  award;  and  (3)  opportunity  for  questions  and 
answers.  The  new,  post-FASA  process,  as  evidenced  by  debriefing  memorandums  at 
Aeronautical  Systems  center,  appears  to  be  much  more  open  and  “user  friendly”  than  the 
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old  process,  though  government  personnel  apparently  needed  some  time  to  adapt  to  the 
new  debriefing  requirements.  Debriefing  memorandums  dated  more  than  six  to  nine 
months  after  FASA,  however,  indicate  that  government  personnel  were  more  fully 
complying  with  FASA  changes.  Based  on  the  debriefing  charts,  scripts,  and  question- 
and-answer  lists,  the  biggest  difference  between  the  old  and  new  processes  appeared  to  be 
the  opportunity  for  unsuccessful  offerors  to  ask  “relevant”  questions  about  the  source 
selection  procedures. 

Investigative  Question  2 

How  do  government  and  contractor  personnel  view  the  differences  between  the 

old  and  new  debriefing  processes? 

Government  and  industry  participants  generally  believed  the  new  process  is  better 
than  the  old  one.  The  findings  indicate  that  government  personnel  now  view  the  post¬ 
award  debriefing  as  a  much  more  important  part  of  their  job.  Contractors,  for  the  most 
part,  appear  to  view  the  new  process  as  more  worthwhile.  Almost  all  participants  agreed 
that  discussion  of  proposal  weaknesses/deficiencies  and  cost/technical  ratings  is  better 
now  than  before.  There  was  also  general  agreement  that  award  rationale  is  more  useful 
and  informative.  Govenunent  and  industry  were  divided  on  the  issue  of  “reasonable 
responses  to  relevant  questions,”  although  all  seemed  to  agree  that  substantive  issues 
beyond  the  successful  offeror’s  proposal  are  now  more  likely  to  be  addressed.  A 
reasonable  conclusion,  based  on  the  interviews,  is  that  the  new  process  is  more  open  and 
relaxed.  The  atmosphere  of  debriefings  is  less  strained.  Perhaps  the  best  description, 
based  on  the  findings,  is  that  the  new  debriefing  process  is  “less  adversarial”  than  the  old 
process. 
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One  conclusion,  based  largely  on  responses  to  interview  question  7,  is  that  the 
new  process  may  be  working  as  well  as  can  be  expected,  given  the  current  laws  and 
regulations.  In  fact,  participants  from  both  sides  expressed  the  general  belief  that 
providing  too  much  source  selection  information  might  do  harm  to  the  competitive 
acquisition  process.  Government  participants  for  the  most  part  said  it  is  important  to 
maintain  the  subjective  nature  of  the  selection  decision  and  to  guard  confidential  and 
proprietary  information  on  both  sides.  Industry  participants  prefer  more  objectivity  in  the 
selection  decision,  but  some  conceded  that  the  current  process  may  be  the  best  obtainable 
under  the  circumstances. 

Investigative  Question  3 

Has  the  new,  expanded  post-award  debriefing  process  helped  contractors  to 

develop  better  proposals  for  future  procurements? 

Based  on  government  responses  to  this  question  alone,  the  answer  is 
inconclusive.  Two  of  the  seven  government  participants  clearly  believed  the  new  process 
helps  contractors  prepare  better  proposals.  Only  one  clearly  said  “no,”  with  most  others 
falling  somewhere  in  between. 

When  government  responses  to  interview  question  2  are  considered,  the 
government  position  shifts  to  the  positive.  Question  2  asked  if  debriefings  clearly 
addressed  proposal  weaknesses  and  deficiencies.  Most  government  participants  answered 
“yes”  to  this  question.  If  proposal  weaknesses  and  deficiencies  are  more  fully  explained, 
it  follows  that  most  contractors  will  learn  from  their  mistakes  and  do  better  in  the  future. 
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All  industry  participants  indicated,  in  some  positive  sense,  that  the  new  debriefing 
process  helps  them  prepare  better  proposals.  The  key  appears  to  be  that  more  information 
leads  to  better  proposals. 

The  preponderance  of  the  evidence  seems  to  fall  on  the  positive  side.  The  new 
process  has  probably  resulted  in  a  general  improvement  in  contractor  proposals.  The 
findings  indicate  that  more  insight  into  the  reasons  behind  government  source  selection 
decisions,  and  improvements  in  explaining  proposal  deficiencies,  give  contractors  a  better 
sense  of  what  the  government  wants  to  see  in  industry  proposals. 

Investigative  Question  4 

What  has  been  the  impact  of  the  new  debriefing  process  on  the  likelihood  of  a 

debriefed  offeror’s  pursuing  a  protest? 

This  is  the  central  question  of  this  study  and,  based  on  the  findings,  it  is  difficult 
to  answer  conclusively.  Three  of  the  seven  government  participants  said  the  expanded 
debriefing  process  has  led  to  a  reduction  in  the  number  of  protests.  Others  expressed 
concern  that  more  information  on  award  decisions  gives  contractors  more  information  on 
which  to  base  their  protests. 

Industry  participants  generally  believed  that  a  better  understanding  of  the  award 
decision  probably  leads  to  a  reduction  in  protests.  At  least  one,  however,  agreed  with 
some  of  the  government  participants  that  more  information  helps  contractors  make  a 
better  case  for  their  protests. 

The  conclusion  from  the  findings  may  be  a  tentative  “yes.”  The  likelihood  of 
protest  has  probably  decreased  as  a  result  of  the  new  debriefing  requirements. 

Admittedly,  this  conclusion  may  be  based  as  much  on  intuition  as  on  the  findings  of  this 
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study.  A  more  definitive  answer  may  have  to  wait  until  both  sides  have  more  experience 
with  the  new  process.  As  one  industry  participant  phrased  it,  protests  should  “go  down 
markedly”  as  the  new  process  becomes  more  ingrained. 

Investigative  Question  5 

What,  if  anything,  can  be  done  to  improve  the  debriefing  process? 

Government  participants  generally  agreed  that  the  new  process  is  acceptable  as  it 
now  stands.  Do  not  tinker  with  a  good  thing.  One  suggestion  that  might  be  worthwhile 
was  for  the  government  to  develop  a  workshop  or  “road  show”  to  explain  the  source 
selection  process  to  industry.  One  industry  participant  also  stated  the  need  for  some  type 
of  training,  especially  in  the  area  of  proposal  evaluation  by  the  government.  Perhaps  HQ 
AFMC/AQ,  the  command  RFP  support  te^un,  is  ideally  suited  to  undertake  this  industry- 
oriented  training  project. 

Most  industry  participants  said  that  knowing  the  relative  ranking  among  their 
competitors  would  help  them  make  better  business  decisions.  SAF/AQC  might  consider 
a  change  in  policy  to  give  contracting  officers  the  option  of  including  the  relative 
rankings  of  all  offerors  in  post-award  debriefings. 

Concluding  Remarks 

Metzger  and  Golden  have  documented  that  the  number  of  protests  filed  at  GAO 
has  decreased  from  2809  in  fiscal  year  1994  to  2529  in  fiscal  year  1995,  the  first  full  year 
of  FAS  A  implementation  (Metzger  and  Golden,  1997:  viii).  This  decrease  is  a  ten- 
percent  drop  in  protests  in  one  year.  There  is  no  guarantee  that  this  reduction  marks  the 
beginning  of  a  downward  trend,  nor  is  there  any  evidence  linking  the  cause  of  the 
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decrease  directly  to  the  effects  of  FAS  A.  One  can  draw  reasonable  conclusions  from  the 
evidence,  however,  and  state  that  FASA  has  probably  played  some  role  in  the 
improvement  in  the  protest  numbers.  The  precise  role  is  impossible  to  determine  from 
this  study. 

It  is  reasonable  to  conclude  that  the  post-award  debriefing  is  an  important  vehicle 
for  government-industry  communication.  Ideally,  it  provides  an  explanation  of  the 
rationale  for  the  award  decision,  exposes  proposal  weaknesses  and  deficiencies,  and 
assures  the  unsuccessful  offeror  that  it  received  fair  treatment. 

FASA  changes  were  designed  to  improve  post-award  debriefings  by  requiring 
government  agencies  to  provide  more  information  on  the  source  selection  process  and  the 
rationale  for  award.  The  theory  behind  the  changes  was  that  better  information  would 
result  in  better  proposals,  and  that  bid  protests  would  be  reduced  as  unsuccessful  offerors 
gained  insight  into  the  government’s  selection  decisions. 

This  study  examined  the  impact  of  the  FASA  changes  on  the  post-award 
debriefing  process  at  Aeronautical  Systems  Center.  Post-award  debriefing  memorandums 
and  related  documentation  were  analyzed  to  determine  the  differences  in  the  debriefing 
process  before  and  after  FASA.  Also,  government  and  industry  personnel  with  extensive 
post-award  debriefing  experience  were  interviewed  to  gain  their  perspective  on  the 
changes  in  the  process  and  to  help  determine  if  FASA  changes  had  their  intended  effects. 

Results  of  the  study  indicate  that,  at  ASC,  FASA  has  improved  the  post-award 
debriefing  process.  The  process  is  less  adversarial  and  more  open.  Study  participants 
generally  agreed  that  FASA  requirements  have  increased  the  amount  of  information 
provided  to  unsuccessful  offerors  in  debriefings.  Extrapolating  from  this  point,  one  can 
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state  generally,  if  not  categorically,  that  more  information  has  led  to  greater 
understanding  of  the  reasons  for  the  source  selection  decision,  better  quality  of  proposals, 
and  less  likelihood  that  the  imsuccessful  offeror  will  pursue  a  protest.  Categorical 
conclusions  await  the  results  of  future  research. 

Suggestions  for  Further  Research 

Future  researchers  might  consider  the  following  related  areas  for  further  study: 

•  A  similar  study  using  larger  samples  and/or  employing  statistical  analysis. 

•  A  comparison  of  the  post-award  debriefing  process  at  different  Air  Force 
locations. 

•  A  comparison  of  the  post-award  debriefing  process  among  the  services  and 
other  DoD  agencies. 

•  A  comparison  of  debriefings  to  determine  if  large  and  small  businesses  view  the 
process  in  the  same  way. 

•  A  comparison  of  the  pre-award  and  post-award  debriefing  processes. 

•  A  comparison  of  government  and  commercial  debriefing  practices. 
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i^pendix  A;  Staff  Sumnary  Sheet  (AF  Form  1768) 
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STAFF  SUMMARY  SHEET 


SURNAME  OF  ACTION  OFFICER  AND  GRADE 


Possum,  Cindy,  Major 

SUBJECT 


SUSPENSE  DAH 


AFIT/LAS 


51211 


Request  for  Access  to  and  Release  of  Source  Selection  Records  After  Contract  Award 


isa  m  IH7 


SUMMARY  “  - - - 

L  Request  Mr.  David  Thomas  be  granted  access  to  post  award  source  selection  records  at  ASC/SYG  Mr 
Thomas  is  a  student  in  the  Graduate  Contract  Management  Program  at  the  Air  Force  Institute  of 
Technology  (AFIT).  He  requires  access  to  post  award  source  selection  records  pertaining  to  DAC  and 
Other  Contractmg  programs.  The  purpose  of  obtaining  this  information  is  to  complete  a  master’s  thesis  on 
the  Air  Force  post  award  debriefing  process. 

2.  This  request  includes  permission  to  copy  records  for  conducting  academic  research.  Confidentiality  will 
be  mamtamed  and  no  source  selection  sensitive  or  proprietary  information  will  be  compromised.  Copies  of 
all  records  will  be  destroyed  upon  completion  of  the  research. 

3.  This  request  is  made  pursuant  to  AFFARS  Appendix  BB,  paragraph  BB-403(b)  and  AFMC  FAR 
Supplement  Appendix  BB,  paragraph  38. a. 

1 4.  Recommend  approval. 


CINDY  POSSUM,  Major,  USAF 
Program  Manager 

Graduate  Contract  Management  Program 
Air  Force  Institute  of  Technology 


^pendix  B;  SAF/AQC  Letter,  13  October  1993 


DEPARTMENT  OF  THE  AIR  FORCE 
WASHINGTON  OC 


U  OCT  1993 


FROM:  SAF/AQC 

1060  Air  Force  Pentagon 
Washington  DC  20330*1060 

SUBJ.  Air  Force  Debriefing  Policy 

*^0"  ALMAJCOM/FOA  (Contracting) 


^  the  Air  Force’,  debriefing 

After  di^sions  with  MAJCOM  representatives  at  the  World-Wide 
Conference,  review  of  the  results  of  AFMC  debriefing  initiatives,  and 
discussions  with  industry  represenUtives,  we’ve  come  to  the  conclusion  that 
we  need  to  be  more  open  and  consistent  in  the  way  we  do  debriefings 
throughout  the  Air  Force.  Therefore,  a  new  AFFARS  supplement  to  FAR 
subpart  15.10  to  cover  debriefing  policy  is  being  develop^.  Until  that 
wbpart  18  published,  the  following  guidelines  shall  be  followed  for 
debriefings  on  all  competitive  acquisitions  that  use  source  selection 
procures  set  forth  in  AFFARS  Appendix  AA  or  BB.  In  addition,  these 
guidelmes  are  encouraged  for  lower  dollar  value  source  selections. 

a.  Contracting  Officers  should  conduct  debriefings  in  an  open 

nonadve^nal  environment  that  encourages  the  frank  exchange  of 
information  about  the  offeror’s  proposal.  However,  care  must  be  taken  not  to 
disclose  anything  during  a  debriefing  that  would  compromise  the  integrity  of 

inforni.a.n  fron.  Mother 

b.  The  contracting  officer  chairs  and  controls  the  debriefing 

However,  the  people  responsible  for  the  actual  evaluations  such  as  the 
&urce  Selection  Evaluation  Board  Chairperson  and  evaluators  shall  provide 
the  specific  evaluation  results  in  the  debriefing,  unless  an  exception  is 
approved  by  the  Source  Selection  Authority  (SSA). 


up™  Ihpir  W  ^''*' 

s^is=ss?s- 

nsk  M«9sment.  Additionally,  contracting  offi«".  if 

ord^it  Sl“  AO^f4" 

consulted  before  releasing  auch  ZlltioT  “““'  ** 

team  should  caucus  before  nmviMinar  *s^on8.  However,  the  debriefing 
in  advance  which  are  eomnSi-r  answers  to  any  questions  not  provided 

of  proprietary  or  classified^iSoi^rion  Ai?“^  Potentially  lead  to  the  release 
consisLt  Juh  “““J* 

J~1?drtt“o^^r^r“^iS“;;;SttJ!!‘^^^  Oc^ionany,  it 

retained  in  the  source  selection  file,  and  b^“re^d“  to  the  oftro^”  ^ 

as  wen  as  ito  ^'tetl'’"l^±",“?"'*'  *  <-f  P™I>°«1  format 

debriefed  only.  You  should  consent  ri^p^tS^M^^th^”*  , 

-»-s:srffix.s:sr,,4s^^  •*■• 

contract  and  th^urce  K^ond^imT’'  »  copy  of  the 

proprietaiy  information  “d 

telephonic  debriefing"?  ae‘^nt?ttang‘offi«™^^^  «  - 
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2^  If  you  have  any  further  recommendations  regarding  debriefing  process 
improvemente,  please  direct  them  to  my  action  officers  for  source^sSlection 


IRA  LKBjIP 
Associate  Deputy  Maleiant 
Secmifv  (Contracttng) 
Assotart  ^9^  (.Acquisjlon) 


\ 
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ppendix  C;  Interview  Cover  Letter  and  Questions 


To: 


DEPARTMENT  OF  THE  AIR  FORCE 

AIR  UNIVERSITY  (AETC) 


Subject.  Interview  Questions — Air  Force  Post  Award  Debriefing  Process 

1.  Thank  you  for  agreeing  to  participate  in  this  research  project,  which  is  directed  toward 
determining  the  effectiveness  of  the  post-award  debriefing  process.  As  we  discussed 
previously,  the  purpose  of  the  interview  is  to  obtain  your  opinion  of,  and  insight  into,  the 
post-award  debriefing  process  as  expanded  by  the  Federal  Acquisition  Streamlining  Act  of 
1994  (FAS A).  I  would  like  to  emphasize  once  again  that  your  responses  will  be  kept 
strictly  confidential.  All  comments  will  be  on  a  non- attribution  basis. 

2.  A  common  complaint  among  government  contractors  before  FASA  was  that  post¬ 
award  debriefings  did  not  provide  enough  information  to  satisfy  unsuccessful  offerors 
concerning  the  source  selection  decision.  Consequently,  many  protests  were  ffled  simply 
because  the  basis  and  rationale  for  the  award  were  not  fully  understood.  The  theory 
behind  this  research  is  that  the  expanded  post-award  debriefing  process  under  FASA 
provides  better  information  than  the  old  debriefing  procedures,  and  therefore  has 
contributed  to  a  decrease  in  the  number  of  protests.  Your  opinions  and  comments  will 
help  us  to  determine  if  the  new  process  is  working  as  envisioned  by  the  architects  of 
acquisition  reform.  In  addition,  your  feedback  may  be  helpful  in  making  reconunendations 
to  improve  the  Air  Force  post-award  debriefing  process. 

3.  Please  review  the  attached  list  of  interview  questions.  I  wiD  call  you  in  a  few  days  to 
schedule  a  brief  interview  that  is  designed  to  last  no  more  than  thirty  minutes.  The 
interview  will  be  open-ended  but  will  follow  the  general  form  of  the  attached 
questionnaire. 

4.  Once  again,  thank  you  for  your  participation  and  cooperation.  If  you  have  any 
questions  regarding  this  research  project,  please  call  me  at  this  number:  (937)  255-7777, 
ext.  2189.  You  may  also  call  my  academic  advisor,  Professor  Andre  Long,  Acting  Head' 
Department  of  Government  Contract  Law,  Air  Force  Institute  of  Technology  Wrieht- 

PattersonAFB,  Ohio  45433.  His  number  is  (937)  255-7777  ext  3146 


DAVID  R.  THOMAS 

Student,  Graduate  Contract  Management  Program 
Air  Force  Institute  of  Technology 


Atch: 

Interview  Questions 


POST-AWARD  DEBRIEFING  INTERVIEW  QUESTIONS 


Government  Questions 


1 .  The  Federal  Acquisition  Streamlining  Act  of  1994  (FAS A)  directed  many  changes  to 
the  government  acquisition  process,  among  them  the  expansion  of  the  post-award 
debriefing  process. 

a.  Have  you  conducted  or  participated  in  a  post-award  debriefing  of  an 
unsuccessful  offeror? 

b.  Has  your  participation  included  experience  both  before  and  after  FASA 
changes? 

2.  FASA  requires  that  post-award  debriefings  address  significant  weak  or  deficient 
factors  in  the  unsuccessful  offeror’s  proposal. 

a.  In  your  opinion,  do  the  current  debriefings  clearly  identify  proposal 
weaknesses  and  deficiencies? 

b.  If  you  have  post-award  debriefing  experience  both  before  and  after  FASA,  can 
you  see  a  difference  in  the  way  debriefings  have  been  conducted  in  this  area? 


3.  FASA  requires  disclosure  of  the  overall  evaluated  cost  and  technical  ratings  of  the 
successful  offeror  and  the  unsuccessful  offeror. 

a.  In  your  opinion,  do  debriefings  clearly  show  the  overall  cost  and  technical 
ratings  of  the  successful  offeror? 

b.  If  you  have  post-award  debriefing  experience  both  before  and  after  FASA,  can 
you  see  a  difference  in  the  way  debriefings  have  been  conducted  in  this  area? 


4.  FASA  requires  the  overall  ranking  of  all  offerors  and  a  summary  of  the  rationale  for 
award. 

a.  In  your  opinion,  do  debriefings  clearly  provide  the  overall  rankings  and  a 
summary  of  the  reasons  for  the  selection  decision? 

b.  If  you  have  post-award  debriefing  experience  both  before  and  after  FASA,  can 
you  see  a  difference  in  the  way  debriefings  have  been  conducted  in  this  area? 
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5.  FASA  requires  reasonable  responses  to  relevant  questions  relating  to  the  conformity  of 
selection  procedures  to  the  solicitation,  regulations,  and  other  applicable  authorities. 

a.  In  your  opinion,  do  Air  Force  personnel  allow  relevant  questions  and  provide 
reasonable  answers? 

b.  If  you  have  post-award  debriefing  experience  both  before  and  2ifter  FASA,  can 
you  see  a  difference  in  the  way  debriefings  have  been  conducted  in  this  area? 


6.  What,  in  your  opinion,  is  the  biggest  difference  between  the  old  post-award  debriefing 
process  and  the  new  debriefing  process? 


7.  Overall,  are  you  satisfied  that  the  current  post-award  debriefing  process  provides  as 
much  information  as  possible,  given  that  government  personnel  must  conform  to 
applicable  authorities  and  not  divulge  confidential  or  proprietary  information? 


8.  Do  you  believe  the  current  debriefing  process  provides  information  that  leads  to  the 
development  of  more  responsive  and  competitive  proposals? 


9.  Compared  to  the  old  process  before  FASA,  do  you  believe  the  current  post-award 
debriefing  procedures  have  reduced  the  likelihood  of  protest  by  an  unsuccessful  offeror? 


10.  Do  you  have  any  specific  recommendations  as  to  how  the  current  process  can  be 
improved? 


POST-AWARD  DEBRIEFING  INTERVIEW  QUESTIONS 


Industry  Questions 


1 .  The  Federal  Acquisition  Streamlining  Act  of  1994  (FAS A)  directed  many  changes  to 
the  government  acquisition  process,  among  them  the  expansion  of  the  post-award 
debriefing  process. 

a.  Have  you  participated  as  the  representative  of  an  unsuccessful  offeror  in  a  post¬ 
award  debriefing  conducted  by  an  Air  Force  contracting  officer? 

b.  Has  your  participation  included  experience  both  before  and  after  FASA 
changes? 


2.  FASA  requires  that  post-award  debriefings  address  significant  weak  or  deficient 
factors  in  the  unsuccessful  offeror’s  proposal. 

a.  In  your  opinion,  do  debriefings  clearly  identify  proposal  weaknesses  and 
deficiencies? 

b.  If  you  have  post-award  debriefing  experience  both  before  and  after  FASA,  can 
you  see  a  difference  in  the  way  debriefings  have  been  conducted  in  this  area? 


3.  FASA  requires  disclosure  of  the  overall  evaluated  cost  and  technical  ratings  of  the 
successful  offeror  and  the  unsuccessful  offeror. 

a.  In  your  opinion,  do  debriefings  clearly  show  the  overall  cost  and  technical 
ratings  of  the  successful  offeror? 

b.  If  you  have  post-award  debriefing  experience  both  before  and  after  FASA,  can 
you  see  a  difference  in  the  way  debriefings  have  been  conducted  in  this  area? 

4.  FASA  requires  the  overall  ranking  of  all  offerors  and  a  summary  of  the  rationale  for 
award. 

a.  In  your  opinion,  do  debriefings  clearly  provide  the  overall  rankings  and  a 
summary  of  the  reasons  for  the  selection  decision? 

b.  If  you  have  post-award  debriefing  experience  both  before  and  after  FASA,  can 
you  see  a  difference  in  the  way  debriefings  have  been  conducted  in  this  area? 
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5.  FASA  requires  reasonable  responses  to  relevant  questions  relating  to  the  conformity  of 
selection  procedures  to  the  solicitation,  regulations,  and  other  applicable  authorities. 

a.  In  your  opinion,  do  Air  Force  personnel  allow  relevant  questions  and  provide 
reasonable  answers? 

b.  If  you  have  post-award  debriefing  experience  both  before  and  after  FASA,  can 
you  see  a  difference  in  the  way  debriefings  have  been  conducted  in  this  area? 


6.  What,  in  your  opinion,  is  the  biggest  difference  between  the  old  post-award  debriefing 
process  and  the  new  debriefing  process? 


7.  Overall,  are  you  satisfied  that  the  current  post-award  debriefing  process  provides  as 
much  information  as  possible,  given  that  government  personnel  must  conform  to 
applicable  authorities  and  not  divulge  confidential  or  proprietary  information? 


8.  Do  you  believe  the  current  debriefing  process  provides  information  that  leads  to  the 
development  of  more  responsive  and  competitive  proposals? 


9.  Compared  to  the  old  process  before  FASA,  do  you  believe  the  current  post-award 
debriefing  procedures  have  reduced  the  likelihood  of  protest  by  an  unsuccessful  offeror? 


10.  Do  you  have  any  specific  recommendations  as  to  how  the  current  process  can  be 
improved? 


Appendix  D:  Government  and  Industry  Participants 


Government  Participants 

Mark  D.  Bennington 
Contracting  Officer 
ASC/RAKP 
(937)  255-5706 

Particia  E.  Blakely 
Contracting  Officer 
ASC/PKW 
(937)  255-9510 

Karen  Brunn 
Contracting  Officer 
ASC/YPKKC 
(937)  255-0767 

Lavvrence  E.  Cooper 
Contract  Negotiator 
ASC/PKWRB 
(937)  257-2 135,  ext.  4548 


Aundair  D.  Kiimey 
Contracting  Officer 


ASC/PKWOV 

(937)  257-5847,  ext.  4194 


James  E.  Leighty 
Contracting  Officer 
WL/PK 

(937)  255-5252 

Alan  B.  Miller 
Branch  Chief 
ASC/PKWT 

(937)  257-5847,  ext.  4242 

Harry  Schafer 
Branch  Chief 
ASC/PKWR 

(937)  257-2135,  ext.  4497 
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Industry  Participants 
William  J.  Fahle 

Business  Development  Representative 
Hughes  Aircraft  Company 
Dayton  Customer  Center 
2970  Presidential  Drive,  Suite  300 
Fairborn  OH  45324 
(937)  429-5429 

Robert  B.  Finch 
President 

SelectTech  Services  Corporation 
325  Regency  Ridge 
Centerville  OH  45459 
(937)  438-9905 

Dion  Makris 

Manager,  Training,  Simulation,  and  Information  Systems 

Lockheed  Martin  Corporation 

2940  Presidential  Drive,  Suite  290 

Fairborn  OH  45324 

(937)  429-0100 

John  N.  Milligan 
Senior  Program  Manager 
Sumaria  Systems,  Inc. 

Dayton  Office 

3 1 64  Presidential  Drive,  Bldg  8 
Fairborn  OH  45324 
(937)  429-6070 

Barret  L.  Myers 
President 
Sytronics,  Inc. 

4433  Dayton-Xenia  Road,  Bldg  1 
Dayton  OH  45432 
(937)  429-1466 
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